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Introduction 



Schools across the nation are working to adapt and improve curricula and teaching practices to 
meet the standards for learning established by states and school districts. In mathematics and 
science education, “standards-based reform” typically means that teachers must plan and 
implement their curriculum and teaching in relation to challenging content standards with high 
expectations for student knowledge and capacities. A major question for education decision 
makers is how best to assist teachers in improving their curriculum content and teaching 
practices, with the ultimate goal of improving student achievement. 

The Council of Chief State School Officers (CCSSO) was awarded a three -year NSF grant in 
2000 under the Research on Learning in Education Program (ROLE) to conduct an experimental 
design study to determine the effectiveness of a new model for professional development aimed 
toward improving the quality of instruction in math and science education. Wisconsin Center for 
Education Research (WCER) and TERC Regional Alliance for Science and Math partnered with 
CCSSO to provide study staff to carry out the project. The professional development model 
focused on developing teachers’ skills to analyze and use in-depth data on the enacted curriculum 
(curriculum actually taught in classrooms), to identify weaknesses or gaps in their instruction in 
relation to state standards, and to revise teaching strategies and curriculum to meet the problems 
identified by the data. Data for a school’s enacted curriculum in a specific subject is obtained 
through a comprehensive survey of all teachers assigned to teach math or science. 

The purpose of the three-year CCSSO study was to design, implement, and test the effectiveness 
of the Data on Enacted Curriculum (DEC) model for improving math and science instruction. The 
model was tested by measuring its effects with a randomly selected sample of “treatment” schools 
at the middle grades level as compared to a control group of schools at the same level. The study 
focused on teachers of middle grades math and science in 5 urban districts with a total of 40 
middle schools comprising the pool for random selection to groups. The enacted curriculum 
model drew on prior development of a Surveys of Enacted Curriculum (SEC) data collection 
instrument and method of analyzing and reporting data on instructional content and practices 
(Blank, Porter, & Smithson, 2001; Porter, Kirst, Osthoff, Smithson, & Schneider, 1993; Porter, 
2002). Recent research studies on effective professional development in math and science 
informed the development of the DEC model (Loucks-Horsley, Hewson, Love, & Stiles, 1998; 
Cohen & Ball, 1999; Love, 2000; Garet, Birman, Porter, Desimone, & Herman, 1999). The 
intervention model used in this experimental design study was built upon four characteristics of 
effective professional development indicated by the research: (a) development activities focus on 
subject content and active learning strategies; (b) activities are coherent with the curriculum 
teachers use; (c) data are used to track continuous improvement and define follow-up steps; and 
(d) teacher development involves school-based collaboration and teacher networking aimed 
toward sharing improvement ideas, practices, and strategies. 
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Summary of Findings: 

What Has Been Learned from the Experimental Design Study of the 
DEC Professional Development Model 



The study findings summarized in this paper will inform education leaders and educators about 
the effectiveness of the DEC data-based model for evaluating and improving instruction. Readers 
can consider the use of the DEC model to help to focus and direct their professional development 
programs, especially the initiatives aimed toward improving quality of instruction in science and 
math. The study results also inform education researchers about the degree to which the effects of 
the model did have significant results on instruction as evaluated through the experimental 
design. The following findings from the study are described in this paper: 

■ How the enacted curriculum data-driven model for professional development was 
designed and implemented. 

■ Methods of effective collection and reporting of enacted curriculum data for use by 
educators. 

■ How enacted curriculum data on alignment of content of instruction are used to 
improve instruction. 

■ How school-level data on instructional practices in a subject are reported and used. 

■ How data on teacher preparation and teacher opinions/attitudes are used to initiate 
improvement activities. 

■ Effects of the DEC model on improving instruction over time. 

■ What was learned about local support and organization for implementing the DEC 
model. 

■ Lessons learned about conducting an experimental design study using randomized 
school-level trials. 

The first portion of the paper outlines the study rationale, prior research, and study design. The 
main body of the paper provides a description of key study findings and highlights key data charts 
and tables of data demonstrating use of the DEC professional development model. 
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Study Rationale, Prior Research, and Design 



A prominent theme of current approaches toward improving instruction is to train educators to 
analyze quantified data on their students’ achievement, as measured by standardized assessments, 
and then to apply these data along with other measures toward improving instruction and student 
learning. Assuming that teachers do effectively use these data to identify problems in 
achievement for their students, how do they then move to analyzing their instruction in relation to 
the data on student achievement? This is the question driving the improvement model tested in 
this study. 

Comparable, reliable data on the curriculum actually taught in classrooms (enacted curriculum) 
allows teachers to analyze their own instruction in relation to standards and in comparison to 
instruction among teachers within a school and in comparison to other schools. These kinds of 
analyses allow teachers to identify where the links between instruction and achievement can be 
strengthened. 

Research Questions 

The CCSSO project involved development, testing, and analysis of the effects of a new data- 
based professional development model designed to improve instruction through applying lessons 
from data analyses. The study tested the model across a range of schools at the same grade level 
serving comparable students. The following three research questions led to the study design 
initiated in 2000: 

■ What are the effects of the Data on Enacted Curriculum (DEC) model for 
professional development for teachers on improving the quality of instruction in 
middle grades math and science? 

■ What is the extent of variation in the content of classroom instruction and variation in 
instructional practices in middle grades math and science, based on teacher survey 
data? 

■ How is the DEC professional development model effectively implemented in large 
urban districts at the school level? 

This paper describes the main findings of the study and summarizes what has been learned about 
how the DEC model for professional development can be implemented in schools and can assist 
educators in improving instruction in science and math. Lessons about the use of an experimental 
design with random assignment of treatment at the school level are discussed. Several other 
papers analyzing the research questions have been produced from this project and are referenced 
in this paper. 

Design for the Study 

CCSSO organized a study team which included professional development experts to lead the 
implementation of the DEC professional development model and outstanding education 
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researchers to lead the evaluation of implementation and effects of the model in schools. The 
DEC model for improving instruction was implemented in five large urban districts: Charlotte - 
Mecklenburg, Chicago, Miami-Dade, Philadelphia, and Winston-Salem/Forsyth County. The 
study sample comprised 40 middle schools located in these districts. All of the math and science 
teachers in treatment middle schools were the target groups for the surveys and professional 
development model. An experimental design was used to measure the effects of the program 
model through comparing the degree of change in instruction in treatment schools to the change 
in control schools. The study design contained the following major steps: 

■ Select urban school districts and middle schools for the project, and orient district 
and school leaders to the project objectives and design. 

■ Collect baseline data (year 1) on instruction in math and science through in-depth 
surveys with all math and science teachers in study schools. 

■ Randomly select 10 schools per participating district into 2 groups for conducting the 
place-based experimental design: (a) treatment schools (schools/teachers receive 
DEC model technical assistance and professional development); and (b) control 
schools (comparable schools with no assistance). 

■ Provide professional development workshops and technical assistance to leadership 
teams and teachers in treatment schools on how to use enacted curriculum data (18 
months). 

■ Conduct implementation research in study sites and validate self-report survey data. 

■ Carry out follow-up surveys (year 3) with science/math teachers in all schools. 

■ Analyze change in teaching practices and determine effects of model. 

■ Provide technical assistance and professional development to control schools. 



The following flow chart illustrates the timing of key steps in the DEC project. The dotted line 
captures the key steps in the three -year time period of the study. 

Chart 1 

Flow Chart of Study Design 




Improved 

Student 

Achievement 
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The project team was led by Rolf K. Blank, director of education indicators at CCSSO. The 
research and data analysis component of the study was led by Andrew Porter, Vanderbilt 
University, and John Smithson, Wisconsin Center for Education Research (WCER) at University 
of Wisconsin-Madison. The implementation of the DEC professional development model was led 
by Diana Nunnaley and Mark Kaufman of the Regional Alliance for Mathematics and Science 
Education at TERC. 

Prior Research and Development: Three categories of studies support the design and testing 
of the study model 

Surveys of Enacted Curriculum. The teacher survey instruments used in the study of the DEC 
model to analyze math and science instruction were previously developed and field-tested by 
CCSSO and WCER through a collaborative project with 1 1 states and 300 schools across the 
states. The instruments were designed by a committee of subject specialists and researchers. A 
portion of the survey items were based on previous studies — including Reform Up Close (Porter, 
et al., 1993), National Survey of Science and Math Education (Weiss, Banilower, McMahon, & 
Smith, 2001), TIMSS teacher questionnaire (NCES, 1996), NAEP background surveys, and state 
and national standards in science and math education. 

The surveys are inclusive of content standards and curriculum currently in place across states, but 
the intent and design of the surveys aim toward collecting objective, reliable data on instructional 
practices and subject content regardless of the intended standards or curriculum for a school and 
its classrooms. That is, the survey items are designed to be neutral and comprehensive in order to 
gain an accurate picture of current instruction as reported by teachers. (For survey development 
process, see Blank, et al., 2001; Porter, 2002.) Surveys are administered anonymously and 
respondents are guaranteed the data will not be used for teacher or school accountability. 

Survey topics. As a result of several stages of design, testing, and survey development work, the 
resulting Surveys of Enacted Curriculum (SEC) for mathematics and science were ready for 
application in the present study to provide reliable, comparable data based on teacher responses 
concerning instructional content and practices in math and science instruction. Additionally, the 
survey items on teacher background, class characteristics, and schools conditions provide the 
capacity for multivariate analyses to explain differences in instruction. The SEC instruments 
produce the following categories of data: 

■ Subject content of instruction (topic by expectations by time) 

■ Classroom instructional practices (general and specific practices by time) 

■ Teacher course preparation, recent professional development, demographics 

■ Teacher beliefs and opinions regarding teaching and school conditions 

■ Class characteristics 

With the survey data, any specific set of state standards or assessments, or local curriculum, can 
be analyzed in relation to the enacted curriculum as reported by teachers. The survey tool serves 
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as an independent, common reference point for analyzing data across schools, districts, and states. 
(See www.SECsurvey.org for further description of survey development.) 

The teacher survey instruments for middle grades math and science were used in year 1 of the 
DEC study to establish baseline data on teaching practices and instruction in the 40 schools 
selected for the study. These schools were randomly selected for treatment and control groups 
after initial data collection. Enacted curriculum data were reported to the treatment group schools 
in year 1, and school teams used these data as the basis for professional development based on 
analysis of instruction across classrooms and schools (see DEC professional development model, 
following). In year 3 of the study, surveys were repeated with teachers in treatment and control 
schools in order to measure the extent of change in instruction. 

Research on effective models for professional development. Educational standards and systemic 
approaches to reform have emerged in the past decade as favored policy tools for promoting 
“world-class” public education for all students. In mathematics and science education, standards- 
hased reform typically means teachers must plan and implement their curriculum and teaching 
practices in accordance with state or district content standards (Leonard, Penick, & Douglas, 
2002). A primary reason for the importance of professional development with current veteran 
teachers is that standards-based education requires teachers to have abilities to both communicate 
basic knowledge as well as develop advanced thinking and problem-solving skills among all 
students (Loucks-Horsley, et al., 1998; Birman, Desimone, Garet, & Porter, 2000). Standards- 
based instruction expands the expected behaviors and methods of teaching that are needed 
(National Research Council, 1995). 

A professional consensus has emerged suggesting that particular characteristics of professional 
development can make it “high quality” or effective in improving teaching and increasing student 
achievement (Garet, et al., 1999; Hiebert, 1999; Loucks-Horsley, et al., 1998; Kennedy, 1998; 
Desimone, Porter, Garet, Yoon, & Birman, 2002; Corcoran & Foley, 2003; Supovitz, 2002). The 
research-based characteristics include the following program design elements: 

■ Active learning opportunities for teachers, responsive to how teachers learn and take 
leadership roles. 

■ Extended duration, sustained over time. 

■ Focus on content, high standards, and how students learn the content. 

■ Collective participation of groups of teachers from the same school or department. 

A key step in the process of moving toward standards-based instruction is evaluating the quality 
of science and math instruction across classrooms to determine what changes and improvements 
need to be made. To provide effective formative evaluation designed to improve quality of 
curriculum and instruction requires reliable, comparable data that allows educators to determine 
the degree of consistency in the curriculum being taught, and then to identify the sources of 
variation in the enacted curriculum, the subject content, and classroom practices (Blank, 2002; 
Porter, 2002). 
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The DEC professional development model is designed to assist school leadership teams and 
teachers to use quantified survey data to identify the subject areas which have a high degree of 
variation in instruction and low student achievement, and then to implement a process of using 
data to focus discussions of leaders and teachers specifically on what is needed to improve 
instruction in these areas (CCSSO, 2002a). The DEC data-driven approach to designing 
professional development is based on research documenting the characteristics of schools and 
learning communities which use effective strategies for improving student achievement 
(Schmocker, 2002; Fullan, 2000; Love, 2000; Wellman & Garmston, 1999). The Standards for 
Student Evaluation emphasize the important purpose for student assessment data of providing 
appropriate and timely feedback to teachers for improving teaching and learning (Joint 
Committee, 2003). The general approach underlying the DEC professional development model is 
for teachers to gain knowledge through use of data: 

...To support those leading mathematics and science education reform at the school or 
district level so that teachers can become inquirers into how to best improve student 
learning. (Love, 2000) 

Enacted Curriculum Data in the Professional Development Model 

Underlying the efforts to develop an effective, reliable method of collecting, reporting, and using 
data on the enacted curriculum is the proposition that useful analysis of student achievement 
results — the usual target of data-driven improvement — needs also to incorporate education inputs 
and processes (Porter & Smithson, 2001; Martin, Blank, & Smithson, 1996; Blank, 2002). 

Education outputs (i.e., assessed measures of student achievement) are mediated and influenced 
by a variety of inputs to and processes in the delivery of instruction. Any useful analysis of the 
performance of our schools requires information pertaining to all three factors — inputs, processes, 
and outputs. 

Education inputs cover factors such as teacher characteristics (e.g., educational background, 
preparation in teaching field, years of experience); student characteristics (e.g., prior 
achievement, attitudes toward school); classroom resources and equipment; school characteristics 
(e.g., school climate and safety, instructional leadership); and parental involvement. Other input 
variables include student racial/ethnic heritage, socioeconomic background, and geographic 
location (urban vs. suburban vs. rural). 

The education processes and activities which actually occur in the classroom (i.e., the enacted 
curriculum) have the most immediate impact on student achievement. Indicators of these 
processes provide a description of the enacted curriculum and are central to any analysis of the 
opportunity to learn. Data on enacted curriculum provide information on what influences the 
curriculum taught by teachers, the instructional practices and cognitive activities in which 
students engage, materials and equipment used in classrooms, classroom organization and roles of 
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students, subject content of assignments students receive, and methods of assessment used to 
measure learning. 

Given these assumptions about the role that data can have in assisting educators, the DEC model 
for professional development focuses on three applications of the data on enacted curriculum: 

1 . Description of current curriculum and classroom practices. 

2. Analysis of differences within and across schools. 

3. Diagnosis of reasons for variation and gaps in subject content and practices. 

Descriptions of the enacted curriculum make it possible to assess the effects of policy on practice 
and provide an opportunity for school-based conversations about curricular goals and practices. 
Knowing the degree to which goals are or are not being met, and having some theories as to why, 
makes fine-tuning and corrective measures possible. 
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Summary Findings from the Study 



The findings from the experimental design study of the DEC model are summarized in eight main 
points. The summaries are intended to inform education leaders and researchers about the 
effectiveness of the model for evaluating and improving instruction. 



Finding 1: Development and Implementation of the 
Enacted Curriculum Model for Professional Development 

In year 1 of the study, CCSSO project staff undertook three major steps to begin the study. First, 
following district and school introductory orientation to the project, baseline surveys were 
conducted with math and science teachers and school administrators in middle schools within the 
target districts. 

Second, the project staff provided details about the organization and design of the DEC model for 
using quantified school-level curriculum data to improve instruction by use of an inquiry-based 
approach to learning and improvement. 

Third, the project staff began implementing the model by working with school leadership teams 
through centrally located workshop sessions and school-based, on-site technical assistance. 



Chart 2 

DEC Professional Development Model to Help 
Schools to Reach Improvement Goals 
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Developing Data Skills of Educators in Year 1. The DEC project plan for implementation of 
professional development and technical assistance in the target middle schools recognized that 
few teachers, or administrators, had been involved in a professional development process for 
analyzing data. The project sought to introduce key ideas and train staff in applying strategies for 
working collaboratively within a group of professionals and to begin to ask tough questions about 
which students were or were not learning, what content was being learned, and why some 
students were not learning. Second, and of equal importance, professional development 
workshops modeled best practices in teaching and provided resources and support which enabled 
participants to engage their own colleagues. The DEC model was presented to the district staff 
and school teams in five participating districts and set forth three specific goals for the work with 
math and science leadership teams and subsequent development with teachers: 

■ Learn to use rich, in-depth data to inform decisions about curriculum practice, 
assessment, organization, and materials. 

■ Gain skills in collecting, analyzing, and displaying data, working collaboratively, and 
organizing data-driven dialogue. 

■ Learn how to set measurable student learning goals, develop data-driven local 
improvement plans, and sustain process. 

In other data-driven school improvement initiatives, such as the Charlotte-Mecklenburg district 
approach to analyzing North Carolina “end of grade” test scores, school groups have undertaken 
the challenge of learning skills for working with student achievement data. However, the DEC 
approach to using instructional and curriculum data is new, and few schools or districts had the 
advantage of previously adding to their analyses the kinds of data generated when an entire 
mathematics or science department completes the Surveys of Enacted Curriculum. 

District and School Responsibilities and Commitments 

Table 1 provides a summary outline of the DEC professional development model as it was 
presented to the participating districts. In year 1 at the orientation stage of the DEC model, the 
CCSSO team outlined the key responsibilities of the districts and schools for effective 
implementation: 

■ Regularly scheduled meeting times for school leadership teams, and time commitment for 
teachers to work on applying the model. 

■ Decision-making support for next steps in schools. 

■ Inclusion of DEC model into school and district professional development. 

■ Access to school data including state assessment results. 

■ Focus on measurable results. 
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Table 1 

DEC Professional Development Model Time- Activity Outline 



Year 1: 


Mar-Apr 2001 


Orientation of district and school leaders; teachers complete 
baseline SEC 




Aug-Sep 2001 


Introductory PD workshop for leader teams (two days); 
develop data skills and begin data inquiry 




Oct-Nov 2001 


Technical assistance in schools to introduce model to teachers 
(leader half-day session each school) 


Year 2: 


Jan-Feb 2002 


PD workshop #2 (one day): Use of content data and 
instructional practices data 




Feb-Apr 2002 


Technical assistance in schools to set school targets for 
improvement based on data analysis 




May 2002 


PD workshop #3 (one day): Analyzing student work and 
comparing instructional strategies 




Sep-Oct 2002 


Technical assistance in schools; work with teachers to apply 
data to instruction 




Nov-Dec 2002 


Evaluate progress toward improvement objectives; re-focus 
efforts within schools 


Year 3: 


Jan-Mar 2003 


Teachers continue work in teams and application of data 
lessons 




Apr 2003 


Complete follow-up surveys with teachers 



Each school selected and organized a leadership team for the DEC model with five to seven 
members representing the following kinds of school staff: principals/assistant principals, 
math/science team leaders or department heads, math/science teachers, data specialists (school 
staff working with data). 

Initial workshop training was geared for the leadership teams and focused on developing data 
skills and inquiry approaches with data. The training provided leaders with a design and sequence 
of activities for working with teachers to bring the model into schools. (For full explanation and 
detail on the DEC model design, see Nunnaley, Using Data on Enacted Curriculum, 2003.) 

What Was Learned about Developing and Implementing the DEC Model — District Level 

In this study, the DEC model was implemented and tested in five large urban districts. The study 
design included research on the implementation process based on data from observations, 
interviews, staff feedback, and administrator surveys. The results of the implementation analysis 
across the five districts produced several central findings regarding effective implementation of 
the DEC model — particularly how the district level actions affected model effectiveness. (The 
findings are described in detail in CCSSO, Year 2 Progress Report, 2002b.) 
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1) District Leadership and Support for the DEC Model 

All five districts indicated strong interest in the DEC model at the district leadership level. 
CCSSO provided information about the newly funded study to states and large districts. The five 
selected districts were similar in expressing commitment to standards-based improvement of 
math and science education as well as to the use of data with schools and teachers to lead 
improvement. Four of the five districts had received Urban Systemic Program grants from NSF. 

Written information and materials were provided by CCSSO outlining the study design, model 
for assisting schools, district responsibilities, selection of schools, and role of project staff in 
working with schools. The same on-site presentations were made by the CCSSO project staff to 
district administrators and school principals at the outset of the project. CCSSO project staff 
worked to maintain a consistent approach to soliciting district leadership and requesting support 
for the DEC model, especially in gaining school-level commitment to the model. Two main 
factors varied in district-level leadership related to the DEC model: communication of support to 
target schools, and change in district priorities for programs. 

Districts varied in giving clear support for the project to target schools. In each district, a 
district contact person was assigned to serve as liaison between the DEC model and the target 
schools. In one district, strong support for the model from top leadership provided a consistent 
message to selected schools that the DEC model was a priority for math and science 
improvement, and the district administrators provided consistent leadership. For example, district 
staff provided early and regular communications to principals in the target schools encouraging 
use of professional development time, scheduling sessions with CCSSO project staff, and 
advocating broad teacher involvement. Conversely, in another district, district contacts for the 
project did not have authority to contact schools directly, and communications of support and 
facilitating arrangements were indirect. In a third district, the primary district administrator for 
math and science went on leave in year 2, and, as a result, initial positive contacts with schools 
were not sustained and di strict-level leadership and support for the model declined. 

District leadership change resulted in new program priorities. In two districts, top-level 
leadership change in the district resulted in declining interest and time for the DEC professional 
development model. In one district, a state takeover of district governance prevented math and 
science staff from having a direct role in communicating with schools and providing support for 
the model with schools. In another district, change in superintendents in year 2 and district-wide 
change to a choice model for student assignment severely reduced the district role in supporting 
the model. 

When district leadership and support for CCSSO project staff was less clear, interactions and 
involvement from school leaders were more problematic — that is, more time was required by 
CCSSO project staff for arranging technical assistance and professional development workshops, 
and less school-wide support was given to the model. 
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2) Integration of DEC Model with District Program for Professional Development 

In two of the project districts, the data-driven DEC model for improving math and science was 
highly consistent and coordinated with the districts’ own math-science improvement initiatives 
and their organization structures for assisting schools and teachers. For example, in Miami, the 
math and science district-level staff were funded and organized to work directly with middle 
schools. The DEC model became integrated into the district’s own school -based approach to 
assisting middle schools and emphasis on improving skills of all teachers. In Winston-Salem, the 
DEC model was strongly advocated by district specialists also responsible for an NSF grant, 
because the model provided important data for formative evaluation and analysis of instruction 
with standards; in this case, the role of the district staff greatly facilitated the work of the CCSSO 
staff with school leadership and promoted teacher participation in activities. 

In a third district, professional development was largely controlled by principals, and the district 
math/science specialists did not operate through a school-based model or have direct contact with 
school leadership. This structure did not facilitate scheduling and interaction with schools for the 
DEC project staff. In a fourth district, in year 2, a program priority was set for all schools to 
improve student performance in reading, resulting in schools devoting more professional 
development time on this subject and decreasing focus on use of the DEC model. 

3) Improving Planning and Implementation in Partnership with Districts 

From the results of the study, CCSSO identified several steps for improving implementation of 
the DEC model at the district level. One step is to plan with the district leaders for how the DEC 
professional development can be integrated into existing professional development. Although this 
goal was verbally presented to districts and schools, CCSSO staff leaders have identified several 
actions which could have been taken to improve the integration process. 

For example, staff observed that each district had a planned set of school- or district-level 
activities for analyzing and applying results from state or district achievement tests. This kind of 
activity could provide an opportunity for demonstrating links between enacted curriculum data 
and student achievement data. Then, both sets of data could be enriched by comparisons and 
implications for instruction. 

In each district, the DEC professional development model plan and 18 -month schedule should be 
delivered to district and school leaders at the outset of the project, and a formal agreement should 
be signed affirming that the plan will be followed. To help to ensure consistent implementation of 
the plan, a local DEC advisory panel should be selected and used. 

DEC is built on a school-based approach to improving instruction, which includes formative 
evaluation of practice through data, developing common instructional practices, analyzing data, 
and building a learning community among staff. The time requirements are organized differently 
from traditional professional development workshops or activities. These differences and 
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assumptions behind the model should be clarified and presented to distinct leaders at the outset 
and refreshed during the project. 



A trained district-level team responsible for DEC model follow-up and assistance to schools 
would greatly increase the probability of integration of the model with local professional 
development programs and schedules. In planning with district leaders, staff time and 
involvement should be planned as to provide a small district team to work at least part time in 
carrying out the model — meeting with school teams to analyze curriculum data, convening 
follow-up sessions in the schools, helping to identify additional resources or training, and 
providing in-school technical assistance. The result would be improved integration of the project. 
CCSSO project staff would provide training to the district team as the work with schools is 
initiated and would provide continuing support thereafter. 
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Finding 2: How SEC Data Were Collected and 
Reported to Target Schools 



In spring of year 1 of the study, CCSSO project staff conducted orientation visits to each school 
where they outlined the rationale for the study, presented a schedule for key steps in the project, 
and reviewed the procedures for administration of the Surveys of Enacted Curriculum (SEC). 
Orientation focused on ensuring confidentiality and use for school improvement (not for individual 
accountability), how to organize group administration in the school to increase response rates and 
decrease turnaround time, and how data will be presented and used to assist schools. 

Survey data collection for the baseline teacher SEC was conducted in spring 2001 (4 districts, 40 
schools). Due to attrition of schools from the study in year 1, 10 additional schools joined the 
study in fall 2002, and additional teacher surveys were administered. 

Response rates. The overall survey response rate in the baseline teacher SEC in year 1 was more 
than 75 percent. The rate did vary widely by school across the 40 sample schools from more than 
90 percent down to 50 percent. A total of 604 teacher surveys were completed for the baseline 
survey. 

At the end of the treatment phase of the project, the follow-up SEC was conducted in all study 
schools (treatment and control), and a total of 439 surveys were completed by teachers, yielding a 
response rate for the year 3 follow-up survey of 55 percent. (See appendix A for Descriptive 
Data on the survey sample.) 



Table 2 

Teacher Surveys of Enacted Curriculum Completed, District by Subject 



District 


Year 1 


Year 3 




Math 


Sci 


Total 


Math 


Sci 


Total 


Charlotte-Mecklenburg 


50 


49 


99 


50 


19 


69 


Chicago 


28 


27 


55 


17 


11 


28 


Mi ami -Dade 


114 


105 


219 


83 


88 


171 


Philadelphia 


34 


31 


65 


24 


17 


41 


Winston-Salem / Forsyth County 


71 


65 


136 


69 


61 


130 


Total 


319 


285 


604 


243 


196 


439 



Steps to increase survey responses. The SEC instruments are extensive and require significant 
effort for completion by teachers. Typically teachers require 60 to 90 minutes to complete the 
entire survey. Given the complexity of the survey tools, the response rates for this study are quite 
good, but they require effort to achieve. The following specific methods of increasing response 
rates were developed in the DEC project: 

■ Administrator and teacher orientation to explain the puipose of the surveys. 
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■ Group administration led by principals or district staff. 

■ Cash or gift certificates for teacher survey completion participation. 

■ Professional development credits. 

■ Dinner or lunch provided at the survey administration site (e.g., hotel). 

■ Follow-up reminders or scheduling through local 

For DEC project staff, encouraging the district 
coordinator to play a strong role was a key step. In the 
fifth district, Winston-Salem, which was added to the 
study in 2002, the district coordinator and his 
math/science staff went to each target middle school to 
administer the survey. The district staff allocated time 
on the school staff schedule — typically after school — 
and obtained full survey participation in each school. 

Preparing data charts. All completed teacher survey 
data were entered into a central database for producing 
pre-designed data graphs and charts to be used in 
professional development workshops with the schools 
in the treatment sample. The complete SEC dataset 
from teacher responses in year 1 (treatment and control 
schools) formed the baseline data for evaluation of 
effects of the model on improving instruction. Each 
treatment school received a customized school-based 
report within three months after initial administration of 
the teacher surveys. The reports contain more than 150 
pages of school-specific data organized into charts 
covering 20 categories of survey results, including 
instructional practices, content of instruction, class 
characteristics, teacher preparation, teacher attitudes, 
and school conditions. (Appendix B includes a list of 
charts used with target schools, and description of 
summary scales). To view a set of charts for a school, 
go to www.SECsurvev.org/tools .) Each category of 
charts was disaggregated with grouping variables: 

1 . Grade level 

2. Class achievement level 

3. Class size 

4. Percent minority 

5. Percent female 

6. Percent limited English 

7. Teacher professional development 



district or school coordinators. 



Interpreting Data Charts 

The SEC survey results are reported and 
analyzed using three formats: item profiles, 
summary scales, and content maps and 
graphs. 

Item Profiles present data from individual 
survey questions, grouped by topic and item 
format (see chart 4-1). The data are shown in 
horizontal bar graphs. The mean is indicated 
by a solid vertical line, and the shaded bar 
represents responses that are one standard 
deviation above the mean and one standard 
deviation below the mean. Generally, the 
responses at the mean and within the bar 
represent about two-thirds of all responses to 
a question. The number of teacher responses 
per group (e.g., middle, elementary) is 
reported in parentheses. 

Summary Scale is an average score for a 
group of five to eight questions in the survey 
centered on a specific concept underlying 
curriculum or instruction, e.g., active learning 
in science (see chart 4-2). Scales are formed 
by purposeful selection of items and statistical 
analysis of responses to determine scale 
reliability (e.g., .81 for communicating math 
understanding). The selected scale items 
typically cut across different sections of the 
survey, and items may have different kinds 
of responses. The scale measures are 
“standardized scores,” meaning the average 
score for the scale for the whole group of 
teachers is set equal to 0, and the standard 
deviation (a measure of variation in responses) 
for the whole group is 1. Scale score 
differences would mean that subgroups of 
teachers (e.g., middle, elementary) differ on 
the concept being measured. 

Content Maps and Graphs. Teachers report 
time spent on subject content during the year 
using a content matrix covering topics and 
expectations for learning. Responses of 
teachers are aggregated by grade level and 
reported with either (a) a content mapping 
program (available in MSExcel), which gives a 
three-dimensional picture of variation in time 
across the whole curriculum (see chart 3-1 
and appendix C, Interpreting a Content Map); 
or (b) a bar graph format, which shows 
average percent of time by topic and 
expectation (see chart 3-6). 
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The core reporting topics for SEC data charts and decisions on disaggregation categories were 
completed in the survey development project in 2001 (Blank, et al., 2001). The data are reported 
to schools in three types of charts — item profiles, summary scales, and content maps and graphs. 

Scales and item profiles are used to report on instmctional practices and teacher characteristics and 
attitudes. Content maps and graphs are used to report data on content of instruction as well as 
degree of alignment between instruction and standards and assessment. (See appendix C for a 
description of SEC alignment procedures; see appendix D for a guide to interpreting content 
maps.) 

While the full set of school data results tended to provide more information than schools required 
for their individual needs, the broad range of data collected ensured that all schools were able to 
find relevant data within their school report which addressed particular areas of concern or 
interest. The training provided to school leadership teams made use of these data as a starting 
point for conversations and activities by the teams during workshops and technical assistance 
visits. 
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Finding 3: How Data on Alignment of Curriculum Content 
Are Used to Improve Practice 

A school receives its SEC data report several weeks after the surveys are completed by teachers. 
The complete report for instruction in math, or science, consists of 25 selected topics from the 
survey sections on instructional practices in the subject, content of instruction, and teacher 
preparation and attitudes. The data for each topic are disaggregated by selected student, class, and 
teacher characteristics, and school level data are reported together with district level data. 

With the alignment content maps and statistics, educators can examine critical differences in 
content of instruction within a school or across schools in a district. Especially pertinent to No 
Child Left Behind and the requirement for identifying “schools in need of improvement,” the 
alignment model provides a method for identifying discrepancies between curriculum being 
taught and the content in standards and assessments used by a state. The curriculum and 
instructional analysis can also be linked to analysis of student achievement results to help 
teachers begin to identify explanations for low performance based on the curriculum. The data do 
not analyze quality of instruction but do clearly demonstrate differences across schools, class and 
student characteristics, and teacher background and preparation. The analysis also helps educators 
to identify areas of the standards which are not being taught, or are taught with only limited time 
or emphasis, and which expectations for learning expressed in standards or assessments are not 
being included in the curriculum. 

The alignment maps, graphs, and other data reporting charts were designed by the CCSSO and 
WCER study team to provide the same variables for disaggregation across all charts and to 
provide comparison of instruction in a school with district and state standards. The series of 
decisions which led to a common reporting format and initial set of topics and methods for 
reporting and disaggregating data were made through advice of a survey development and 
reporting committee (during the 1998-2001 development process). In reporting data, each district 
selected the specific types of data disaggregations and comparisons that were desired to match the 
local plans for using data. (See appendix D for illustrative alignment content maps displaying 
state standards for each DEC study district in comparison with district-level teacher survey data 
results for content of instruction in math and science.) 

Reporting Math Instruction Aligned with Math Standards 

Instructional data in the SEC content charts are reported using a matrix format based on a two- 
dimensional view of content — i.e., content topics and expectations for student learning. (To 
review the survey instruments, go to www.SECsurvev.org/tools .) 

■ Main topics in middle grades math: Number/operations, measurement, algebraic 
concepts, geometry, data/statistics/probability 

■ Expectations for learning: Memorize/recall, perform procedures, demonstrate 
understanding, conjecture/proof, solve non-routine problems 
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In the SEC data collection phase, all teachers reported on the percent of time the target class spent 
on instruction over the course of an academic year. Chart 3-1, for example, displays the amount 
of time grades 6 through 8 teachers in Miami study schools spent teaching math by topic and, for 
those topics taught, the degree of emphasis on different expectations for learning. The contour 
map allows educators to quickly learn which topics and expectations were emphasized in 
instruction, and the degree to which the instruction varied by main topics (number, measurement, 
algebra, geometry, etc.). 

In addition, teachers can compare their instruction in the di strict with the standards defined by the 
state, both by topic and by expectations for learning. This level of “alignment analysis” using 
visual comparison of content of instruction does not require statistical analysis, which is very 
conducive for use by a variety of audiences and users. However, the alignment analysis process 
includes an overall statistic of degree of alignment, or “alignment index,” which varies from 0 to 
1 (Porter & Smithson, 2002). For example, the following is based on the year 1 survey results and 
coding of the Florida state standards: 

Alignment index for grade 6 math (treatment schools): 

Degree of alignment between instruction and Florida state standards, grade 6 = .19 

(scale from 0 to 1). 

Professional development for educators with the DEC model began with training in data skills 
including how to analyze and apply the enacted curriculum data charts and how to interpret 
differences in the color contour maps and bar graphs signifying high and low emphases of 
instruction across a school or district. The school team began the analysis process with one chart, 
using their experience of completing the survey and their existing knowledge of instruction in the 
school. 

Three-step process. Educators were asked to go through a three-step process — predict, 
observe/analyze, and interpret. Teachers were asked first to predict what they will see in the 
degree of consistency, or alignment, between math instruction and district and state standards (for 
example Florida middle math standards). In step two, educators looked at the charts for math 
instruction and standards for their district and state. Educators worked together in teams to share 
what they saw — which topics and expectations had high emphases of time and how consistent 
they were with the standards. For example, chart 3-1 shows real data from Miami study schools. 
Miami math educators could see that instruction in grades 6 through 8 strongly emphasized 
number sense (where the green chart colors represent higher time commitments), while the 
Florida standards emphasized number sense, measurement, and data analysis for grade 6 
instruction. 

School level. In step three, educators refered to a chart which analyzed the content in their school, 
such as charts 3-2.1 and 3-2.2, which compare math instruction in two schools, Richmond and 
Homestead, grades 6 through 8, with further comparison to Florida standards. Teachers compared 
and contrasted instructional content in their school with another school and the average for the 
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district in comparison with standards. Thus, teachers could begin to analyze their school’s 
instruction as compared to the overall district average instruction and compared to state standards, 
which provide broad goals for learning. Teachers also could compare instructional content from 
the survey to content in a district curriculum guide, if available. In the Homestead-Richmond 
example, the results show that Homestead teachers placed greater emphasis on all five 
expectations for student learning than did Richmond teachers. 

Drill down by topic. Another approach to using the data is to “drill down” to look at instruction 
on specific topics in the curriculum. Chart 3-3 displays Miami middle grades math instruction on 
number sense. This topic typically receives a high degree of emphasis in the early elementary 
grades, and given the demand on time for the middle grades math curriculum, a useful analysis 
can be made on the question of how much time is spent in grades 6 through 8 on number sense. 
(Critiques of U.S. math curriculum focus on excessive repetition of topics across the grades.) 

In chart 3-3, the specific instructional topics are disaggregated in detail at the subtopic level, 
which is the level at which teachers reported the data. Subtopics are useful for educator team 
analysis because they likely correspond to the organization of the teacher’s lesson plans and 
course outlines. The results show that Miami teachers emphasized numbers related to operations 
in all three of the middle grades, while the Florida standards placed emphasis on patterns, 
exponents, factors, and estimation. 

Grade by grade. A next step school teams took in using the enacted curriculum alignment data 
was to consider the vertical alignment of instruction by grade, sometimes called curriculum 
articulation. For example, in the Homestead school data example from year 1, shown in charts 3- 
4.1 and 3-4.2, the teachers looked at the degree to which time spent on instruction in grade 6 is 
then repeated in the same pattern across topics in grades 7 and 8. Note that grade 8 instruction 
continued to have a high percent of time on number sense and little time on algebra or geometry. 
Teachers and administrators can use these data to dig deeper by comparing instruction for specific 
classes and by asking their school instructional staff several questions: 

■ Is the vertical alignment pattern from grade to grade present with only certain classes, 
or is it a problem across a school or district? 

■ What are reasons for topics being repeated from one year to the next? For example, 
are teachers responding to poor student performance? Or, are they responding to a 
large number of students entering from other schools (who are not well prepared)? 

■ Are books and materials vertically aligned? 

■ Is the instruction by grade consistent with state standards by grade? 

Instructional content data in relation to assessments. Another way to view the survey data 
reported by schools is to compare data on instruction with the content being tested in state 
assessments. Charts 3-5.1 and 3-5.2 display data for two Miami middle schools in comparison to 
the content on the Florida Comprehensive Assessment Test (FCAT) in mathematics for grade 8. 
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The FCAT was content-coded using the SEC coding procedures. (See Appendix C for 
procedures; Porter & Smithson, 2002.) Three points can be highlighted: 

■ The instructional content maps for both schools show high consistency in topics 
taught in middle grades math with the topics on FCAT, grade 8. As discussed above 
for chart 3-1, time on instruction is concentrated heavily on number sense/properties, 
with much less time allocated to topics of measurement, algebra, geometry, and 
data/analysis. 

■ The expectations for math instruction by Miami teachers are inclusive of all five 
types of expectations, while the FCAT assessments focus almost solely on the 
expectation of perform procedures. 

■ The correlation of alignment of grade 8 math instruction and the FCAT math 
assessment = .22 (school average; 0 to 1 scale). 

Educators in the Miami schools in the DEC project used these findings to improve their 
instructional strategies and curriculum design and cited the data in planning professional 
development. 

Histogram method of reporting. Charts 3-6. 1 and 3-6.2 present a second method of displaying 
and reporting the SEC instructional content data for two schools through a more traditional 
histogram. As opposed to the three-dimensional contour map format, the histogram display of 
data provides further details about the differences in instruction across topics and expectations. 
One advantage of the histogram is the capacity for comparing the row or column percentages at 
the margins, such as total percent of class time on number sense (row one), or percent of time on 
perform procedures (column two). These sums allow educators to compare the total allocation of 
time by topic. By comparison, the contour maps display the whole picture of the curriculum, 
particularly the intersection of topics and expectations for learning. 

Schools Chose a Target Content Area or Expectation to Improve 

A key step in the DEC model for improving instruction is selection of a specific set of priorities 
for improving instruction based on initial analysis of the data. The school leadership teams 
selected targets for focusing on content areas needing improvement, generally during the first or 
second DEC professional development workshop. In the DEC project, 18 of the 20 treatment 
middle schools did identify one or two specific areas of math or science content or instruction to 
be the focus of a school, department, or grade-level team. Schools identifying content topics 
typically also identified a subgroup of students (e.g., African American females, students below 
proficiency). Content targets also were associated with goals focused on expectations for learning 
(i.e., higher-level thinking skills, problem solving, applications). 
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Table 3 

Example Improvement Targets (Defined Year 1): 
Three Treatment Schools, Miami-Dade 



School 


Target for Improvement 


School #87 


* Measurement, algebraic thinking 

* Nature of science 

* Analyzing information 

* Hands-on learning / active learning 


School #810 


■ Measurement, geometry, measurement in science 

■ Analyzing curriculum alignment to FL state assessment for skills 
needed and coherence across the grades 

■ What is taught compared to what is learned 

■ Student-centered learning goals; student understanding of student 
scores; students working cooperatively 


School #85 


* Seek three-percent improvement in bottom quartile students 

* Improve student achievement 

* Scientific thinking 

* Measurement, graphing 
■ Spatial sense 



Case Study: How DEC Model Was Used to Improve Math Instruction 

Diana Nunnaley, professional development director for the DEC project, has written a more 
detailed description of how one school in the Miami-Dade school district used the DEC 
professional development model as the core strategy for improving instruction in mathematics. 
This case example shows how the DEC process of using curriculum data worked well to intersect 
with the district and school priorities for building a learning community and focusing on data as a 
vehicle for change. (See “Test Scores: What Can They Tell Us?” from Hands-On, TERC, June 
2004, at http://www.ccsso.org/proiects/Surveys of Enacted Curriculum/Products/ .) 

Case Study: Use of Data to Improve Science Instruction in Winston-Salem Schools 

A second case description of how schools used SEC data to analyze science instruction was 
provided in a recent article in Science Teacher. The article presents a summary of how schools in 
one district applied their science instructional content data. (See Blank & Hill, January 2004, at 
http:// www.SECsurvey.org/products .) 
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Chart 3-1 

Mathematics Curriculum Content 
Miami - Dade Public Schools 

Instruction / Assessment 



Miami-Dade Public Schools 




E 

E 

o 

CJ 



Florida Grade 6 Mathematic 
Content Standards 




E 

E 

o 

O 



Percentof Instruction 




Data on Enacted Curriculum Study: Summary of Findings 



23 



Chart 3-2.1 

Mathematics Curriculum Content 
Miami-Dade Public Schools 
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Chart 3-2.2 

Mathematics Curriculum Content 
Miami-Dade Public Schools 
Instruction / Standards 
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Chart 3-3 

Mathematics Content Map AAI 
Number Sense/Properties/Relationships 



Miami-Dade Public Schools 
Mathematics Instruction 





































































f 














1 














} 
























































f 




























i 









































































B C D E F 



Place Value 
Whole numbers 
Operations 
Fractions 
Decimals 
Percents 
Ratio, Proportion 
Patterns 
Real numbers 

Exponents, scientific notation 
Factors, multiples, divisibility 
Odds, evens, primes, comp. 
Estimation 
Order of Operations 
Relationships betwn. operations 
Mathematical Properties 



Florida Grade 6 

Mathematics Content Standard 













































































7 


























4 


► 

























B C D E F 



Percent 

Instruction 

( 1 %ofInstr. 



1 . 0 +% 

0.75-0.99% 

0.5-0.74% 

0.25-0.49% 



2 class periods) 0.05-0.24% 
Lessthan0.05 % 




4+% 



3-3.9% 



2-2.9% 



1-1.9% 



Percent 

Assessment 

Score 



0.01-0.9% 



< 



LessthanO.Ol % 



B Memorize 

C Communicate Understanding 
D Perform Procedures 
E Conjecture/Prove 
F Non-routine/Novel Problems 



26 



Data on Enacted Curriculum Study: Summary of Findings 



Chart 3-4.1 

Mathemaitcs Content Map AAI 
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Chart 3-4.2 

Mathematics Content Map AA4 
Miami-Dade Public Schools 
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Chart 3-5.1 
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Chart 3-6.1 
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Chart 3-6.2 
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Data on Enacted Curriculum Study: Summary of Findings 



Finding 4: How Data on Variation in Instructional Practices 
Are Used to Improve Instruction 

In the DEC study, schools and districts used the data on classroom and school differences in 
instructional practices in two ways: 

■ To gain a broad view of types of practices being used in a school or district, or 

■ To analyze in-depth an issue or problem identified through curriculum content 
analysis (as described in finding #3). 

In the SEC instrument, teachers reported on the percent of total class time in which each of 15 to 
20 possible instructional practices for the subject were used during a school year. Teachers 
reported on instruction for the same class and period of time as reported on for subject content of 
instruction. 

Chart 4-1.1 shows data from the survey results for the main types of instructional practices in 
science. The item-level results show the percent of time reported by practice with data aggregated 
across teachers by district and school. The statistics in chart 4-1.1, using the floating bar 
approach, display the mean percentage and the distribution of reported practices one standard 
deviation above and below the mean. This graphic approach to reporting data allows the teacher 
or administrator to quickly see the degree to which multiple practices are present and vary among 
teachers and classrooms. For example, in Winston-Salem middle schools, science teachers 
averaged 20 percent of time on lab activities and 12 percent of time on collecting information 
about science. The average science teacher devoted only 6 percent of time for learning activities 
outside the classroom or lab. These categories of instructional practices are very broad, and they 
are primarily useful for gaining a general overview of how science is taught in a school or district. 
The extent of variation in practices provides key information which educators can use to 
determine how instruction is organized and delivered across classrooms, and then to develop 
ideas about how instruction can be focused and improved to attain standards-based goals for 
learning. 

A second way that the instructional practice data are used by schools is through relating practices 
to content alignment and student assessment results. Chart 4-1.2 displays science instructional 
practices by district and school level with data disaggregated by student achievement level. This 
display of data allowed Winston-Salem educators to ask the question, Does instruction in science 
differ for students with lower prior achievement as compared to students with higher 
achievement? The surveys include a section of items on class description, including the teacher’s 
report of the achievement level of the class. Thus, one strategy for improving instruction for all 
students is to use this kind of chart to launch a discussion among teachers around the following 
kinds of questions: 

■ What instructional activities were used with higher-achieving students vs. lower- 
achieving students? What assumptions can be made about each practice? 

■ How effective are these practices? 

Data on Enacted Curriculum Study: Summary of Findings 
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■ Are some of these practices worthy of consideration for raising the performance of 
lower-achieving students? And if so, what is needed? 

The instructional practices data can also be compared with recent student assessment results (e.g., 
prior end-of-year tests, end-of-quarter tests), which are reported by class or grade. The DEC 
schools typically examined state assessment results for their schools in relation to the alignment 
of instruction with specific content topics (as described in finding #3). However, differences in 
instructional practices across classrooms can also be viewed from the perspective of student 
assessment results, as in the following example: 

Are students with higher test scores found in classes . . . 

■ With more hands-on, active learning or with more teacher-led activities? 

■ With more frequent in-class testing and/or multiple methods of testing? 

■ With more participation in small group work or with more individual work? 

■ With greater use of educational technology during instruction? 

Scales Summarize Instructional Practices in Relation to Standards and Policy Goals 

Chart 4.2 provides a display of six scale scores (i.e., average for a set of items from the survey), 
which focus analysis on core concepts of instructional improvement found in state and national 
standards. (See Appendix A for a list of items by scale.) For example, the results for Winston- 
Salem science at the district level show that on average science classes spent more than 60 
percent of instructional time on active methods of learning (e.g., lab activities, collecting 
information, working in small groups, and not sitting and listening to the teacher). 

At the same time, another scale score on scientific thinking shows science classes in the district 
and school spent relatively little time on student scientific thinking (i.e., activities in which 
students design investigations or experiments or draw their own conclusions). There are several 
possible reasons for a low score, and further analysis may reveal that the group of items ask about 
activities which were not expected in the local curriculum or may have been conducted in many 
classes but did not require large amounts of class time. The instructional activities being surveyed 
do not all require equal amounts of time. 

Two of the scales in chart 4.2 — use of multiple assessments and use of educational technology — 
indicate that teachers demonstrated a high degree of variation in practices at both the district and 
school level. The scales provide a summary view of the methods of improving instruction. 
However, the degree of variation leads to closer examination of responses across the items in the 
scale and comparison of approaches to assessment and technology among teachers: 

■ Are differences among teachers causing wide variation? 

■ Are teachers using different types of assessments in their classes? 

■ Are teachers using technology differently in instruction? 
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Chart 4-3 provides more detailed analysis of practices during lab activities/experiments. The data 
in this chart display item results concerning teacher reports about what students were doing in a 
lab activity. This specific chart for Winston-Salem science shows data for all schools surveyed 
(elementary, middle, and high), and the data are broken out by grade level. Teachers completed 
these items after they had reported the total amount of time spent on lab activities in science 
instruction. Several of these items are included in the scale of active learning and several in the 
scientific thinking scale. The data in chart 4-3 show several major differences in how lab 
activities operated in middle schools, with differences by grade. For example, grade 8 teachers 
reported widely varied use of step-by-step directions in conducting lab activities and wide 
variation in students making predictions or hypotheses. 

Chart 4-4 gives more details about the use of educational technology, including lab equipment in 
science instruction. In charts 4-1.1 and 4-1.2, data from the “large-grain items” on use of time 
showed that educational technology was used in science classes an average of 20 percent of time, 
across teachers. The survey focused further on specific types of technology through a 
representative sample of technology used in science teaching, as displayed in chart 4-4. 

All of the science technology use varied widely at the district and school level. Running water 
and other equipment in science classes varied from rarely used to weekly use. Computer lab 
interfacing and electrical outlets varied in use from rarely to bi-weekly (7 to 36 times). In 
addition, nearly one -third of teachers, particularly at grade 6, reported some of the technology 
was not available. 
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Chart 4-1.1 

Instructional Activities in Science 

By Grade Level 
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Chart 4-1.2 

Instructional Activities in Science 

By Level of Student Achievement 
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Chart 4-2 
Science 

Scale Measures of Instructional Practices 

By Grade Level 
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Chart 4-3 

Laboratory Activities During Science Instruction 

By Grade Level 
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Chart 4-4 

Use of Calculators, Computers & Educational Technology in 

Science 

By Grade Level 
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Finding 5: How Data on Teacher Preparation and 
Teacher Attitudes Are Used with Schools and Teachers 



The SEC results provide comprehensive data for analyzing teacher preparation, both quantitative 
course data and qualitative data on teacher views on their conditions for teaching and their own 
need for improvement. The study identifies three areas in which these data are used to assist 
schools and districts in focusing instruction and curriculum improvement efforts: 

1 . Quality of current professional development, 

2. Prior course preparation of teachers in their teaching fields, 

3. Teacher beliefs about teaching and views about teaching conditions. 

Analyzing Quality of Professional Development Provided to Teachers 

In the DEC project, teachers reported baseline data in year 1 on the amount of professional 
development they received in the prior 12 months in their assigned teaching fields (math or 
science), the focus or emphasis of the professional development activities, and characteristics of 
the activities as compared to the four criteria of quality: coherence, active learning, content 
focused, and collegial participation. 

Charts 5-1.1 and 5-1.2 provide a summary analysis of the professional development activities of 
science teachers in the study. Chart 5-1.1 shows that more than two-thirds of middle grades 
science teachers in the study schools received professional development on teaching to standards 
or implementing the curriculum for their district, and a majority of teachers participated in the 
study of science content. In addition, the data show that more than 80 percent of the teachers who 
participated in activities focused on standards or new curriculum reported that the activities 
positively met the four criteria of quality. 

Chart 5-1.2 provides a trend analysis of change in the amount of time on professional 
development of science teachers, by type of activity, over the two-year period of the DEC study. 
The trend analysis shows that science teachers had substantial increases in professional 
development in four areas: standards, implementing curriculum, in-depth study of content, and 
networking with colleagues. 

Charts 5-2.1 and 5-2.2 provide summary data on the professional development of mathematics 
teachers in the study. Chart 5-2.1 provides data on activities by teacher response to the quality 
criteria. More than 80 percent of the mathematics teachers participating in these activities 
responded positively that the activities met the four criteria of quality. Chart 5-2.2 graphs the 
extent of change in the professional development of math teachers over the two-year study. Math 
teachers had substantial increases in three areas: assessment strategies, standards, and in-depth 
study of math content. 

Charts 5-3 and 5-4 display trends in teacher professional development from year 1 to year 3 of the 
study for Miami-Dade. Chart 5-3 shows change in the percent of middle grades science teachers 
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in school 51 who participated in professional development. The data are reported by type of 
activity and by grade level of teachers. In this school, professional development increased 
substantially for teachers in grade 7 in the area of implementing standards and increased in 
implementing new curriculum and new teaching methods for grade 8 teachers. Overall, 
professional development regarding the needs of all students and in-depth study of science 
content declined for teachers in school 5 1 . 

Chart 5-4 identifies through bar graphs the trends at the district level in types of professional 
development activities experienced by middle grades math teachers. Users report the bar graphs 
provide ease in viewing trends over time for percentages of teachers at each level of activity, 
while floating horizontal bars in chart 5-3 facilitate viewing the variation in responses. 

These data are useful for district leaders in analyzing the degree to which initiatives aimed toward 
improving teacher knowledge and skills are working. For example, chart 5-4 shows that in year 3 
of the study almost 20 percent of math teachers received more than 35 hours of professional 
development on implementing standards in math instruction, which was an increase of 5 
percentage points from year 1. The percent of math teachers receiving 16 to 35 hours of 
professional development on standards also increased about 5 points. The percent of teachers 
participating in professional development on multiple assessment strategies and using teacher 
networks increased substantially from year 1 to year 3. 

Data on Teacher Course Preparation Used to Identify Teacher Needs 

An important function of SEC data for decision makers and professional development providers 
is to analyze specific content background weaknesses or strengths of teachers. For example, 
analysis of current alignment of instruction to standards may show that teachers are not spending 
sufficient time in classrooms on geometry or statistics/probability. Information on the course 
preparation of math teachers can show whether they are well prepared in these math subject areas. 
Analysis of these data might also show that teachers have a strong math background but little 
preparation for how to teach the content to their assigned grade levels. 

Chart 5-5 displays the preparation of Miami middle grades teachers in math based on the number 
of college courses completed in the subject. The district summary shows that the average math 
teacher in grades 6 through 8 in the Miami schools sample had taken three to four “refresher” 
math courses, one or two advanced college math courses, and about five math education courses. 
Of interest to district leaders was, first, the high variation in teacher preparation among grades 6, 
7, and 8, with grade 8 teachers holding about twice as much course preparation in math as other 
teachers. Second, the teachers in the sample reported wide variation in level of preparation (as 
indicated by the width of the colored bar, which represents two-thirds of responses). The colored 
bar shows that more than one-third of teachers in the schools had less than two courses of 
preparation in math or math education. The column on the right shows that the average math 
teacher at school 8 1 1 (a DEC treatment school) had less preparation than an average teacher in 
the district sample. 
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Chart 5-6 displays data on the science course preparation of Miami science teachers. The district 
summary shows that the average Miami middle grades science teacher in preparation had taken 
seven to eight life science courses, five to six courses in physical science, three to four Earth 
science courses, and three to four science education courses. The colored bar is wide at each 
grade level for both the district total and the target school indicating a high degree of variation in 
science preparation across both the district and the school. Some teachers had a strong 
background in science while others with the same subject assignment had a poor science 
background. The level of preparation for science teaching is about the same at each grade 6 
through 8. 

The data can be extremely informative for district and school leaders in focusing and planning 
teacher professional development to target development in specific content areas based on 
specific needs. In addition, the data on course background can be combined with the data on 
teacher-reported instructional practices to guide improvement for teachers whose instruction does 
not match standards and whose course preparation has been minimal. 

Teacher Attitude Data to Examine Beliefs about Teaching and Views on Teaching 
Conditions 

Charts 5-7 and 5-8 report data for one district on teacher beliefs about teaching math/science and 
teacher opinions about conditions for teaching. The DEC project has demonstrated that data from 
the more subjective teacher attitude and opinion items of the survey are extremely important for 
analyzing differences in approaches to instruction in math and science. The DEC professional 
development model advances the idea of building a learning community among teachers and 
administrators within a school. Staff regularly communicate about their instructional strategies, 
content, issues, and needs. The analysis of data from SEC results for a school and district are an 
excellent vehicle for pursuing the learning community objectives. The data on teacher attitudes 
and opinions are particularly valuable as a way to begin conversations concerning ideas about 
teaching their subjects, views about how students learn, and reactions to conditions for teaching. 

The first six items in chart 5-7 illustrate the use of the data on teacher beliefs. The chart is 
presented to teachers in a school data analysis session, often prior to examining other charts on 
instructional practices or subject content of instruction. The data results show divergent views on 
several instructional strategies, including order of learning, use of calculators, basic skills, and 
challenging math content. Recognizing the divergent views of teachers within a school provides 
an excellent opportunity for discussion about how views on these topics relate to math 
instruction, use of different teaching strategies, and teaching approaches for students with 
different backgrounds. 

The data in chart 5-8 address issues of teaching conditions and collegiality — that is, how teachers 
see their roles as faculty of the school. The items report teacher views on working with peers, 
involvement in decisions, rules of the school, and access to materials. The results for this district 
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show widely differing views, from positive to negative about conditions for teaching. The results 
in these charts were used by districts to begin data-focused analysis of instruction with teachers. 

...One urban district program director stated that his district found the DEC model 
provided a “three-legged stool of data” that was essential for improving instruction based 
on (a) differences in current instruction, (b) degree of alignment with standards, and (c) 
teacher attitudes about teaching their subjects (S. Hill, Winston-Salem, 2004). 

A major advantage of a data-focused discussion among teachers is that the data can serve as the 
common target for discussion, thereby avoiding situations in which individual teachers have to 
argue and defend their views in front of colleagues. The data clearly show there are differences in 
views and assumptions about teaching a subject and about children learning. Comparing 
differences through group data can lead to further detailed examination of reasons behind the 
data, and discussion of how the views that were expressed affect instruction and efforts to 
improve learning. The goal of using the teacher attitude data is for teachers to learn more about 
underlying issues which might contribute or detract from building a learning community and 
might inhibit frank and open discussion of the reported data about instruction. 
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Chart 5-1.1 



Professional Development of Middle Grades Science Teachers (4 urban districts): 
Topic by Time by Quality Indicators, 2003 (N=127) 
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Legend: 

Time: Clock hours teachers reported spending on specific topic of professional development in prior year. 

Quality Indicators: Four teacher-reported measures of the quality of the professional development activity (teachers reporting 
some time). 

Collegial Participation: Attended with a group of teachers from their school or district. 

Content Focus: The professional development activity had a focus on content knowledge. 

Active Learning: The professional development activity engaged teachers in active forms of learning. 

Coherence: The activity was associated, integrated or coordinated with other professional development offerings. 

Source: CCSSO, Data on enacted Curriculum Project, 2003 
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Chart 5-1.2 

Professional Development of Middle Grades Science Teachers (4 urban districts): 
Topic by Time, N=185, Year 1 (2001) and N=127, Year 3 (2003) 
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Professional Development of Middle Grades Math Teachers (4 urban districts): 
Topic by Time by Quality Indicators, 2003 (N=197) 
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Legend: 

Time: Clock hours teachers reported spending on specific topic of professional development in prior year. 

Quality Indicators: Four teacher-reported measures of the quality of the professional development activity (teachers reporting 
some time). 

Collegial Participation: Attended with a group of teachers from their school or district. 

Content Focus: The professional development activity had a focus on content knowledge. 

Active Learning: The professional development activity engaged teachers in active forms of learning. 

Coherence: The activity was associated, integrated or coordinated with other professional development offerings. 

Source: CCSSO, Data on Enacted Curriculum Project, 2003 
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Chart 5-2.2 

Professional Development of Middle Grades Math Teachers (4 urban districts): 
Topic by Time, N=21 9, Year 1 (2001 ) and N=1 67, Year 3 (2003) 
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Chart 5-3 

Professional Development in Mathematics 

By Grade Level 
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Chart 5-4 

Professional Development of Miami Middle Grades Math Teachers: 
Topic by Time, N=1 1 0, Year 1 (2001 ) and N=81 , Year 3 (2003) 
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Chart 5-4, continued 

Professional Development of Miami Middle Grades Math Teachers: 
Topic by Time, ISM 10, Year 1 (2001) and N=81 , Year 3 (2003) 
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Chart 5-5 

Teacher Course-Taking in Mathematics and 
Mathematics Education 

By Grade Level 
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Chart 5-6 

Teacher Course-Taking in Science and 
Science Education 

By Grade Level 
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Chart 5-7 
Mathematics 

Teacher Opinions (Part 1): 

Beliefs About Student Learning and Professional Collegiality 

By Grade Level 
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Chart 5-8 
Science 

Teacher Opinions (Part 2): 

Beliefs About Student Learning and Professional Collegiality 

By Grade Level 
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Finding 6: Measuring Change in Math and Science 
Instruction over Time 

A primary research goal of the DEC study was to measure the change in math and science 
instruction over time and, specifically, to determine the effects of the DEC professional 
development model on improving alignment of instruction with standards. A main advantage of a 
longitudinal study which collects data in the same schools at two points in time is the ability to 
measure the effects of specific improvement initiatives. The DEC study has the additional 
advantage of comparing this change in two groups of schools randomly placed into treatment and 
control groups. 

Key Findings on Effects of DEC Model on Improving Instruction 

Results from analysis of the effects of the DEC model on improving instruction of math and 
science teachers in the treatment schools are reported in a separate research paper (Porter & 
Smithson, November 2004, forthcoming). The main points of the effects analysis can be 
summarized as follows: 

■ The DEC model did improve quality of instruction, as measured by increasing 
alignment with state standards, when comparing instruction in treatment schools to 
control schools; however, the effects are contingent on the level and effectiveness of 
implementation within the treatment schools. 

■ Schools with a high level of participation in DEC activities showed greater increases 
in alignment of instructional content with state standards than did other schools. 

■ Teachers who served on leader teams for their schools and had high stability in 
participation with the DEC model had greater gains in alignment of instruction than 
did other teachers in the treatment schools. 

■ Due to high teacher mobility in the study middle schools and other leadership and 
organizational changes in schools and districts from year 1 to year 3 of the DEC 
study, only one -fourth of the teachers in the study who completed the baseline 
teacher survey in year 1 also completed the follow-up SEC in year 3. This high 
attrition factor reduced the rigor and statistical significance of the trends analysis for 
DEC model effects. 

■ The DEC data analysis produced an index of alignment between state assessments 
and state standards for each of the study sites, as well as index of alignment for 
instruction. The alignment statistics and content maps for state assessment and 
standards are an important analysis product of the DEC study. (See Porter, 2004, and 
Smithson, 2004, alignment analysis presentations. For a complete listing of alignment 
analyses by state, see www.SECsurvey/ collaborative.) 

Illustrations of Use of SEC Data to Analyze Change in Instruction 

The following examples illustrate how schools use SEC data charts as indicators of change in 
instruction across a set of schools. 
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Change in instructional practices in math. Charts 6- 1 and 6-2 provide data for tracking change 
in instruction in two schools in the same district. Data results for school 87 (treatment school) are 
shown in chart 6-1, and data for school 83 (control school) are shown in chart 6-2. For both 
schools, two years of survey results are displayed, the following examples of which indicate 
change in mathematics instruction: 

■ Instructional time on problem solving and use of manipulatives : One area of change 
from year 1 to year 3 was time on problem solving in math — school 87 math classes 
increased time on problem solving activities (18 to 20 percent on average), while 
school 83 decreased time (24 to 20 percent). Teachers in school 87 also reported 
increased time in use of manipulatives to teach math (10 to 15 percent of total class 
time). 

■ Activities in small groups: Treatment school 87 teachers reported less time on 
activities in small groups from year 1 to year 3. At the same time in control school 
83, use of small groups increased from 12 to 20 percent of time. 

■ Degree of variation by classroom in math instruction practices: Both schools 
reported less variation in instructional practices in year 3 than in year 1 of the study. 
The variation in instructional time (shown by length of the bar) for school 87 teachers 
decreased in collecting data, use of portfolios, use of manipulatives, problem solving, 
and use of small groups. School 83 showed a similar pattern of decreased differences 
in instructional practices among the teachers. 

Change in science instructional practices. Charts 6-3 and 6-4 provide data reported by teachers 
on science instruction in school 51 (treatment school) and school 53 (control school). 

■ Portfolios and small groups increase: From year 1 to year 3, school 51 showed more 
time devoted to use of portfolios in assessing science learning and greater use of 
small group activities. 

■ Greater variation in instruction: In school 53, teachers reported a higher degree of 
variation among classes in time spent on lab activities and investigations in year 3 
than in year 1 , while they reported less time in use of small group activities in year 3. 

Change in alignment of math instruction with standards. Charts 6-5 and 6-6 illustrate content of 
math instruction in two Miami-Dade schools in year 1 and year 3. The instructional data are 
displayed in comparison to the FCAT state math assessment for grade 8. (Note: Instruction can 
also be compared to state standards, as shown in charts 3-1 through 3-5.) 

The FCAT state math assessment emphasized all five main math topics: number sense, 
measurement, data analysis, algebraic concepts, and geometric concepts. The expectations 
dimension places high emphasis on perform procedures, with some emphasis on memorize and 
communicate understanding. 

■ Focus on specific topics: Data in charts 6-5 and 6-6 show that math instruction in 
year 1 in grades 6 through 8 in both schools 87 and 83 strongly emphasized number 
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sense (i.e., green colors show higher average time reported) and some emphasis on 
the four other topics. Expectations for learning in number sense emphasized all five 
expectations. 

■ Differences between schools: In both years of the survey, school 87 teachers reported 
more instructional time on algebra and geometry than did teachers in school 83. 

■ Change over two years in math content: In year 3, school 87 teachers reported an 
increase in expectations for conjecture/prove across several topics, slightly less time 
on data analysis/statistics, and continued emphasis on algebra and geometry. School 
83 teachers reported more instructional time on geometry in year 3. 

These aggregated charts for schools by year are intended to demonstrate how trends can be 
analyzed. As shown in the charts in finding #3, the alignment data can be further analyzed by 
looking at the content maps for specific topic areas (number sense, algebra, etc.). The overall 
patterns for a school can be examined more carefully by looking at the detailed data for each 
topic. 

Change in alignment of science instruction with standards. Differences in science instruction 
content between year 1 and year 3 are displayed in charts 6-7 and 6-8. School 51 (treatment 
school) and school 53 (control school) are both in the Winston-Salem district. The science 
instructional data can be analyzed in relation to the SAT 9 science assessment used in the district. 

The SAT 9 science assessment for grade 8 focuses heavily on content in the topics of life, 
physical, and Earth sciences and nature of science. The expectations dimension focuses heavily 
on students learning to perform procedures and analyze information. The following are highlights 
of charts 6-7 and 6-8: 

■ Teachers in both schools 51 and 53 reported more emphasis on life science in year 1, 
and distribution of emphasis across all content areas except nature of science in year 
3. Teachers in year 3 reported slightly more emphasis on measurement in science 
(possibly reflecting the emphasis on analysis of information in the expectations for 
the SAT 9). 

■ Teachers in both schools reported relatively equal levels of emphasis on all five 
expectations for science content learning; whereas, the SAT 9 places emphasis on 
expectations for memorize and analyze information. 

The content maps allow teachers and leaders to analyze content taught by main topic and subtopic 
by expectations in relation to assessment or standards. (Charts 6-7 and 6-8 show main topics by 
assessment.) Schools receive a report of their instruction at the main topic and subtopic level, as 
well as charts for the average across all surveyed teachers in the district at the same grade and 
subject. Individual teachers can request a report of their own data. Teachers and leaders can 
review how they reported content topics by expectations and discuss their responses with other 
teachers — both to check interpretation of the survey items and terms, and then to further identify 
instructional content differences. 
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Chart 6-1 

Instructional Activities in Mathematics 
Treatment School, Year 1 and 3 
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Chart 6-2 

Instructional Activities in Mathematics 
Control School, Year 1 and 3 
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Chart 6-3 

Instructional Activities in Science 
Treatment School, Year 1 and 3 
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Chart 6-4 

Instructional Activities in Science 
Control School, Year 1 and 3 
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Chart 6-5 

Mathematics Curriculum Content 
Treatment School, Year 1 and 3 
Instruction / Assessment 
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Chart 6-6 

Mathematics Curriculum Content 
Control School, Year 1 and 3 
Instruction / Assessment 




FCAT Grade 8 Assessment 




b 

r 




o 




64 



Data on Enacted Curriculum Study: Summary of Findings 



Chart 6-7 

Science Curriculum Content 
Treatment School, Year 1 and 3 
Instruction / Assessment 



Middle School Teacher Reports (10) SAT- 9 Science Assessment 
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Chart 6-8 

Science Curriculum Content 
Control School, Year 1 and 3 
Instruction / Assessment 
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Finding 7: What Was Learned about Implementing 
The DEC Model in Middle Schools 

Findings from the study address how the DEC model was implemented in the target schools and 
demonstrate lessons learned from effective implementation. Multiple sources of data were used to 
analyze implementation of the DEC model, including surveys with teachers and administrators, 
interviews with local leaders and participants, observations by study research staff, and 
discussions with professional development staff. 

Gaining Teacher and Administrator Buy-In 

The first step in the DEC model which requires full participation by the target group of school 
staff is completion of the teacher SEC instrument. Finding #2 presents what was learned about 
effective steps toward survey administration to produce a high response rate and increase the 
quality of responses. Districts and schools demonstrated different levels of commitment and 
support for the project, especially in encouraging full and willing teacher completion of the 
survey. Essentially, school leaders need to be convinced that the survey completion step will 
produce data that will be useful and relevant to an important improvement goal or program 
initiative in the school. Whereas district leadership and support is an important ingredient for 
success, the survey and data production step is critical for schools to buy in to the DEC model. It 
was learned that schools need clear, timely information on how the DEC model and data will 
assist the schools directly, and why the initial baseline survey is critical to begin the process (and, 
in year 3, how the follow-up survey will measure improvement progress). Teacher support is an 
essential ingredient for school-based data-driven models to succeed. 

Low numbers of responses from the initial baseline survey with teachers produced a severe 
handicap on the subsequent steps in the DEC professional development process, because the 
school leadership team and DEC leadership staff did not have the demonstrated buy-in from 
teachers and had relatively little data to turn around for use in the process. At the end of the initial 
baseline survey administration, four schools were dropped from the study by CCSSO due to low 
response rates (less than 20 percent of teachers completed surveys), and three additional schools 
decided not to continue at the end of the initial professional development workshop in year 1. 
(For details on school status, see Year 2 Progress Report (2002).) Thus, by the end of study year 
1, almost 20 percent of the original sample of schools did not continue due to teacher buy-in 
issues. 

Extent of Participation in Professional Development Activities 

The DEC model relied heavily on a turn-key or training-of-trainers approach. With the turn-key 
model, each school selected a school leadership team whose members were expected to attend all 
professional development sessions (both multi-school sessions and in-school technical 
assistance). The leadership teams then lead other school staff through similar data-driven 
instructional improvement learning experiences. The participation of school leadership teams in 
the 18 -month DEC professional development process is summarized in table 4. 
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Table 4 

School Level Participation in DEC Professional Development 
and Technical Assistance Sessions 



District 


School 

ID 


Total 

Sessions 

Attended 


Average 

Staff 

Per 

Session 


Percent 
Sessions 
Attended by 
Admin. 


Participation 

Index 


Charlotte 


62 


4 


4.0 


75% 


44.4% 




66 


5 


4.8 


60 


32.0 




68 


4 


3.8 


75 


41.7 




610 


3 


7.0 


100 


41.2 




611 


3 


4.7 


100 


93.4 


District 

Average 


NA 


3.4 


4.7* 


79%* 


44.0%* 


Chicago 


91 


6 


5.7 


83 


56.7 




92 


5 


2.2 


60 


73.3 




93 


4 


3.3 


100 


54.2 




97 


4 


3.3 


75 


46.4 


District 

Average 


NA 


4.8 


3.7* 


79%* 


55.3%* 


Miami 


80 


6 


4.8 


66 


48.0 




85 


5 


3.4 


80 


68.0 




87 


6 


4.0 


33 


44.4 




88 


4 


2.8 


75 


55.0 




810 


5 


4.4 


100 


55.0 




811 


6 


5.0 


100 


41.7 


District 

Average 


NA 


5.3 


4.2* 


75%* 


49.7%* 


Philadelphia 


70 


5 


6.4 


100 


58.2 




72 


5 


4.2 


100 


38.2 




76 


5 


2.6 


60 


65.0 


District 

Average 


NA 


5.0 


4.4* 


87%* 


50.8%* 


Winst.- Salem 


50 


7 


3.3 


0 


65.7 




51 


6 


3.2 


83 


52.8 




52 


6 


3.3 


33 


66.7 




56 


7 


2.4 


100 


60.8 


District 

Average 


NA 


6.5 


2.2* 


54%* 


59.5%* 


Overall 

Average 


NA 


4.95 


3.95* 


73 %* 


50.2%* 



*Overall average weighted to reflect number of sessions attended by schools. 
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Professional development sessions. The first dimension of participation is the total number of 
sessions attended by one or more members of a school’s leadership team. The total number of 
sessions attended by schools ranged from 3 to 7, with an average of 4.95 sessions attended per 
school. As outlined in Table 1, sessions included project overview, professional development 
session #1 (multi-school), technical assistance session #1 (in-school), professional development 
session #2, technical assistance session #2 (in-school), and professional development session #3. 

Variation between schools indicates differences in implementation at the school level. As few as 
three sessions were held by a few schools in Charlotte and Philadelphia, one of which started the 
process late due to the school’s decision to wait until new state testing was implemented before 
beginning any data-driven improvement activities. The data also reveal uneven school 
participation in Chicago and Miami schools. For example, school 93 — probably the most engaged 
and committed DEC school in Chicago — did not attend professional development session #3 
because it conflicted with other end-of-year school-wide activities. Although Miami school 87 
participated in six professional development/technical assistance sessions and was one of the 
higher rated implementing schools, two of the sessions were short (half-day) repeat sessions 
conducted with a newly constituted leadership team. Restarting the initiative at school 87 became 
necessary when administration and staff turnover resulted in near total attrition of the original 
school leadership team. 

Average staff participation per session. The second dimension of participation is the average 
number of persons attending DEC professional development sessions as part of a school’s 
leadership team. The average attendance indicates the degree to which a school has full 
involvement from teachers, administrators, specialists, and department heads, as defined by the 
DEC model. School averages ranged from 2.2 to 7 persons, with an overall weighted sample 
average of 3.95 persons. The districts varied in average staff participation per school: Charlotte 
4.7 persons, Chicago 3.7, Philadelphia 4.2, Miami 4.4, and Winston-Salem 2.2. 

Principal participation. The third dimension of participation is the percent of DEC sessions in 
which school leadership teams included the principal or assistant principal. This measure 
indicates the degree of leadership knowledge about the operation of the model and how it can 
assist math and science teachers in curriculum improvement. Charlotte (70%) stands out for 
having a high rate of consistency of administrators present at DEC professional development 
workshops and meetings. Other districts varied widely in school leader participation: Miami 
averaged 75 percent; Philadelphia 87 percent; Chicago 79 percent, and Winston-Salem 54 
percent. One Miami school that changed principals several times during the 2001-02 school year 
had a school leader present for only one of six DEC professional development/technical 
assistance sessions; two other Miami schools had leader representation at all of their sessions. 

Stability of participation. The fourth dimension of school participation in DEC implementation is 
captured in the participation index. The index reflects the average rate of consistency of 



Data on Enacted Curriculum Study: Summary of Findings 



69 




participation among the individuals in the school who attended one or more professional 
development sessions, with 100 percent being the maximum and 20 percent being very low. The 
index (overall average 50.2%) provides a summary measure of the degree to which schools send 
staff to the sessions who can build their skills from one session to the next, and thus have greater 
commitment to the model and capacity for passing on the approach to other teachers. 

School Conditions for Effective Implementation 

In table 4, the data on level of participation by schools across the five district sites shows that 
school-level implementation of the DEC model was extremely varied. Winston-Salem and Miami 
had high numbers of school-level sessions and teacher participation, and more formal sessions 
organized at district and school levels than other districts. Treatment schools in these districts had 
a higher degree of involvement with professional development, and thus had a higher probability 
of teachers using the data skills and analysis results for improving performance (for further data 
on analysis of implementation process, see CCSSO, Year 2 Progress Report, 2002b). 

Analysis across the districts using the range of information available showed that three factors are 
crucial for a high degree of participation by schools and effective implementation of the model. 

1. Allocation of sufficient time for professional development activities in math/science, and time 
allotted specifically for the DEC model. 

Schools with higher rates of participation and depth of implementation also had more time 
allocated by administrators for professional development or had organized teachers for 
special sessions devoted to the DEC data-based approach to improving instruction. Several 
schools devoted after-school time to informing staff, organized special Saturday sessions 
when time was tight, and in a few cases established a whole day for DEC data analysis and 
implementation sessions. 

2. School-level leadership that linked the DEC model to other professional development 
activities. 

One of the outstanding features of schools with more effective use of the DEC model was 
active leadership by the principal and other school leaders to ensure continuity among school 
leadership teams. One of the goals for the teams with higher stability and leadership 
involvement was to move the learning from the DEC model into other instructional 
improvement activities with teachers. Effective schools with this model did not view the 
work through DEC activities as separate, isolated events required by an order from the district 
level. They tended to recognize the important relationship between data analysis skills 
concerning curriculum and instruction and other efforts to analyze strengths and weaknesses 
revealed in student achievement data. They also were able to link the DEC data analysis to 
other curriculum improvement activities, such as improving staff understanding of state 
standards for their subjects and grades or selecting new instructional materials and providing 
staff training for use of new materials. 
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3. Identifying and communicating targets for improvement from DEC analysis. 

The goal of the DEC model is to have all teachers in a school gain skills in curriculum 
analysis and increase their capacity for making decisions about instructional improvement 
based on shared information with colleagues. An important step in the process is moving the 
training of school leadership teams on to the remaining math and science teachers. In the 
current project, this step was largely in the hands of the schools. One strategy that could help 
in communication of the DEC analysis is to focus on specific content targets as a stalling 
point for work with all teachers. The analysis of model effects over time showed that target 
areas identified by the school were typically the content areas that did experience change in 
instruction. Also, DEC project staff found that it was important that the teachers begin 
learning the data analysis skills with a small number of curriculum data charts — that is, to 
start with some identifiable differences in a specific instructional area. 

Another strategy for improving communication of the DEC analysis process to all staff is to 
identify areas of high differentiation in the curriculum. Using data to identify clear gaps 
between teachers in how instruction is delivered causes discussion about reasons for these 
gaps. First, staff can analyze how they responded to the surveys as they did, and then they can 
move on to identifying the relationship between survey responses and real differences in how 
instruction is organized in the school. In the course of these discussions, teachers will begin 
to recognize how ongoing use of these data could be useful for their own decisions about 
curriculum improvement. 
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Finding 8: Lessons Learned on Conducting Randomized 
Experimental Design Studies at the School Level 



The DEC study design used random assignment of schools to treatment and control groups for the 
purpose of determining effects of the model with a scientifically rigorous design. In a prior paper, 
Andrew Porter presented some of the initial study design decisions and effects of local district 
context on the experimental design (2003). At the end of the three-year process, the CCSSO study 
team has identified four major factors that negatively affected the chance for effective 
implementation of the experimental design as initially planned. 

Change in district leadership and control. During this study, two districts experienced 
significant policy changes which had two effects: (a) change in program priorities, which reduced 
the schools’ need to participate in the DEC professional development model; and (b) change in 
the chain of command so that program offices could not directly work with individual school 
principals and staff. 

Teacher mobility. High rates of teacher mobility in the 40 sample middle schools had a strong 
impact on the study design and implementation. From year 1 to year 3 of the study, one-half of 
the teachers of math and science in the sample changed schools (i.e. over a period of two school 
years). Of 660 math and science teachers ( treatment and control) in study schools in year 1, 
only 49 percent were in the same school and subject assignment in year 3 (based on teacher 
assignment rosters collected from five school districts). The high rate of teacher mobility had 
several significant effects on the study design and implementation. First teacher mobility 
decreased the continuity of professional development from one session to the next and from one 
year to the next. Staff had to spend more time devoted to beginning orientation and training a new 
group of teachers, or teachers who were not present at the outset never became involved. Second, 
the measurement of change from year 1 to year 3 proved to be very problematic. Teachers in year 
3 (many of them new since the baseline survey in year 1) either chose not to take the survey, or 
baseline data were missing for purposes of trends analysis. 

Participation rates of teachers. In several districts in year 1 of the study, the lack of support from 
district or school leadership or poor communication to faculty resulted in low baseline survey 
response rates in 20 percent of the sample schools. Due to low rates of teacher buy-in, several 
treatment and control schools were dropped from the study. Project staff did recruit replacement 
schools within the first year of the study; however, increased expense and time for the study were 
required to maintain the expected number of schools. In light of these challenges, the rigor of the 
experimental design could be questioned. 

Problem of treatment vs. no treatment design. Random selection at the school level into two 
groups — one where the treatment model is delivered and a second (control) where no treatment is 
provided — was extremely difficult to communicate and to sell to school systems and made it 
difficult to sustain support for the model. School leaders expressed significant concerns with the 
treatment/control design (which is the common approach to experiments at the individual student 
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or teacher level). The solution this study provided was to offer delayed delivery of treatment to 
control schools after the initial two-year period of the study — however this approach required 
project staff to convince assigned control schools to participate in data collection with no reward 
or gain from the project for two years. In addition to hampering good relations with some districts 
and schools, the solution strained project resources. 

In future applications of a school-level random selection design, use of one of the following two 
designs is strongly recommended: (a) assign schools to one of two or more alternate treatments, 
such as different professional development models, with effects of the different models measured 
and compared over time; or (b) design a study in which the control group is “blind” and analysis 
of dependent variables are conducted with data centrally collected from all schools. 
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Conclusions 



Teachers, administrators, and policymakers across the United States continue to base efforts 
toward educational reform on content standards, a movement in public K-12 education which has 
continued for a decade. At national, state, and local levels, educators are now working to identify 
methods of aligning curriculum and instruction with standards. School leaders seek models for 
professional development which focus on upgrading teacher knowledge and skills to teach toward 
standards. The Data on Enacted Curriculum (DEC) project led by CCSSO has addressed a central 
question in the push for standards-based instructional improvement: How do we fairly and 
accurately determine the status of current instruction across classrooms and schools, and 
measure progress of improvement using reliable data? And, then, how can the data be used to 
inform efforts to guide instructional improvement toward greater alignment with standards? 

The study tested the DEC model for collecting, analyzing, and using data to guide instructional 
improvement in math and science education. The model was tested in a sample of 40 middle 
schools in 5 urban districts. The study findings show the model can be effectively used by school 
teams to lead instructional improvement. The findings further show that when the DEC model 
was effectively implemented in schools, school leaders and teachers used the enacted curriculum 
data as a tool for focusing instruction on content contained in state standards. 

The study also showed that several local conditions are crucial for the DEC model to be effective 
locally. First, a large proportion of math and science teachers in a school must complete the 
baseline and follow-up surveys, so that data analysis and applications of the results have strong 
buy-in from the staff, ensuring the data are fully representative of instructional practices across a 
school. Teachers and leaders need to see an important connection between curriculum data, 
analyses of student achievement, and other professional development in math and science. 

Second, the instructional data need to be presented in displays and graphs which facilitate use of 
the data by teachers and other professionals to compare instructional content and practices with 
standards and assessments. Data workshops need to be scheduled to allow educators sufficient 
time to work together to gain skills in data interpretation and application to their own efforts in 
improvement. 

Third, school and district leaders need to advocate for data-driven strategies for improvement and 
purposely schedule time when teachers can work together to review their instructional data; 
where they can jointly identify gaps in relation to standards and wide variation in content of 
instruction — both topics and expectations for students; and where they can then collaborate on 
strategies for moving instruction toward content contained in state standards and assessments. 

Fourth, professional development initiatives need to be aligned, coherent, and coordinated, so that 
available time is used wisely and so that educators work toward a common improvement goal. 
Follow-up efforts need to build from prior experience and data should be used to track the degree 
of change. Leadership from district and school levels is key for maintaining a coherent and 
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consistent model. The DEC sites with consistent local leadership and support for the model 
showed evidence of strongest gains on aligned instruction. 

The findings of the three -year experimental design study demonstrate that the DEC model has 
strong potential for further application and effective use in schools. The results also show that 
much has been learned about planning, organization, and support, which are key ingredients for 
the model to be effective, and these findings can be used by program developers to improve the 
model and gain positive outcomes. The study results also inform the work of researchers and 
program evaluators who are planning applications of the Surveys of Enacted Curriculum design 
and procedures for analyzing effects of professional development initiatives on improving 
alignment of instruction with standards and assessments. 
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Appendix A - Descriptive Data on Schools, Teachers, and Classes Participating 



Numbers of Districts 








5 




Teachers Responding to Survey 




Mathematics 


Science 


(whole or in part) 




Year 1 


Year 3 


Year 1 


Year 3 






289 


239 


261 


194 


Class reported by Teacher 




Mathematics 


Science 






Year 1 


Year 3 


Year 1 


Year 3 




Grade 2-5 


20 


12 


11 


4 




6 


91 


80 


87 


69 




7 


89 


77 


77 


60 




8 


85 


66 


82 


61 




Total 


285 


237 


257 


194 


Teaching Time 




Mathematics % 


Science % 


(hours/week) 




Year 1 


Year 3 


Year 1 


Year 3 




Less than 4 


10.7 


9.6 


15.7 


35.1 




4-4.9 


22.1 


15.9 


25.3 


43.8 




5 or more 


67.1 


73.6 


57.5 


20.1 



Achievement Level of Students Mathematics % Science % 

Year 1 Year 3 Year 1 Year 3 
High 13.8 15.1 9.2 13.9 

Average 31.8 31.8 31.0 24.7 

Low 35.3 29.7 25.3 24.7 

Mixed 17.3 22.6 33.3 36.6 
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Appendix A - Descriptive Data on Schools, Teachers, and Classes Participating 



Teacher Characteristics 


Mathematics % 


Science % 


Highest Degree 


Year 1 


Year 3 


Year 1 


Year 3 


BA/BS 


59.8 


57.7 


54.0 


54.1 


MA/MS or higher 


40.1 


41.0 


41.4 


41.2 


Other 


0.3 


1.3 


0.8 


2.1 


Major: Bachelors 


Year 1 


Year 3 


Year 1 


Year 3 


Elementary Ed. 


34.6 


31.4 


28.0 


31.4 


Middle Ed. 


10.6 


10.9 


7.3 


10.9 


Math Ed. or Science Ed. 


9.3 


7.9 


5.4 


7.9 


Mathematics/Science field 


10.1 


12.6 


25.7 


12.6 


Combined Education and M/S field 


6.6 


3.3 


4.6 


3.3 


Other field 


28.8 


35.1 


22.2 


35.1 


Major: Highest Degree 


Year 1 


Year 3 


Year 1 


Year 3 


Elementary Ed. 


57.1 


20.1 


22.2 


23.7 


Middle Ed. 


5.9 


7.9 


7.3 


13.9 


Math Ed. or Science Ed. 


7.3 


10.0 


8.4 


9.8 


Mathematics/Science field 


4.2 


4.2 


11.1 


28.9 


Combined Education and M/S field 


4.2 


5.4 


5.4 


6.2 


Other field 


20.1 


19.7 


15.7 


19.6 



Teacher Demographics Mathematics % Science % 





Year 1 


Year 3 


Year 1 


Year 3 


Female 


76.8 


75.7 


66.3 


70.1 


Male 


23.2 


22.6 


31.0 


26.8 


White 


48.4 


53.6 


53.6 


50.5 


Minority 


52.6 


46.4 


37.9 


45.8 
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Appendix B 



SEC Reporting: List of pre -formatted charts for reporting data collected by Surveys of Enacted 
Curriculum (online or paper versions) 

Mathematics Charts 

A: Scale Measures of Instructional Practices 

B: Scale Measures of Teacher and School Characteristics 

C: Class Description 

D: Use of Class Time during Most Recent Unit of Instruction in Mathematics 
E: Use of Homework in Mathematics 
F: Instructional Activities in Mathematics 
G: Problem Solving Activities during Mathematics Instruction 
H: Small Group Work in Mathematics 
I: Use of Hands-On Materials in Mathematics 
J: Assessment Strategies in Mathematics 

K: Use of Calculators, Computers, & Educational Technology in Mathematics 
L: Professional Development in Mathematics 
M: Influences on Instructional Practice in Mathematics 
N: Teacher Course-Taking in Mathematics and Mathematics Education 
O: Teacher Readiness (Part 1) 

P: Teacher Readiness (Part 2) 

Q: Teacher Opinions: Beliefs about Student Learning 
R: Teacher Opinions: Beliefs about Professional Collegiality 
S: Teacher Demographic Characteristics 
T: Content Maps of Instruction & Assessment/Standards 
U: Content Graphs of Instruction & Assessment/Standards 
AA: Fine Grain Content Maps 

Science Charts 

A: Scale Measures of Instructional Practices 

B: Scale Measures of Teacher and School Characteristics 

C: Class Description 

D: Use of Class Time during Most Recent Unit of Instruction in Science 
E: Use of Homework in Science 
F: Instructional Activities in Science 
G: Laboratory Activities during Science Instruction 
H: Small Group Work in Science 
I: Collecting Information in Science 
J: Assessment Strategies in Science 

K: Use of Calculators, Computers, & Educational Technology in Science 
L: Professional Development in Science 
M: Influences on Instructional Practice in Science 
N: Teacher Course-Taking in Science and Science Education 
O: Teacher Readiness (Part 1) 

P: Teacher Readiness (Part 2) 

Q: Teacher Opinions: Beliefs about Student Learning 
R: Teacher Opinions: Beliefs about Professional Collegiality 
S: Teacher Demographic Characteristics 
T: Content Maps of Instruction & Assessment/Standards 
U: Content Graphs of Instruction & Assessment/Standards 
AA: Fine Grain Content Maps 
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Data on Enacted Curriculum Study 



Science Year3 Scale Items Reliability Coefficient Scale Mean 

Science Scales (reliability if item dropped) (item mean) 



Communicating Scientific Understanding 


csu 


0.761 


1.69 


Q36 


Write about Science 




0.751 


1.57 


Q47 


Complete written assignments from the textbook or workbook. 




0.731 


1.30 


Q49 


Talk about ways to solve science problems. 




0.729 


1.77 


Q51 


Write results or conclusions of a laboratory activity. 




0.706 


2.02 


Q56 


Organize and display the information in tables or graphs. 




0.716 


1.76 


Q64 


Display and analyze data. 




0.722 


1.68 


Active Learning in Science 


AcLS 


0.715 


1.62 


Q31 


Collect data or information as part of science (as part of science 
homework). 




0.679 


1.63 


Q34 


Collect information about science. 




0.694 


1.50 


Q37 


Do a laboratory activity, investigation, or experiment in class. 




0.674 


2.13 


Q38 


Work in pairs or small groups. 




0.695 


2.02 


Q39 


Do a science activity with the class outside the classroom or 
science laboratory. 




0.693 


0.88 


Q42 


Use science equipment or measuring tools 




0.678 


1.92 


Q43 


Change something in an experiment to see what will happen. 




0.689 


1.40 


Q52 


Work on an assignment, report, or project that takes longer than 
one week to complete. 




0.694 


1.47 


Student Reflection on Scientific Ideas 


RPS 


0.686 


1.39 


Q35 


Maintain and reflect on a science portfolio of their own work. 




0.637 


1.04 


Q53 


Work on a writing project or portfolios where group members help 
to improve each other's (or the group's) work. 




0.604 


1.06 


Q55 


Ask questions to improve understanding. 




0.671 


1.82 


Q58 


Discuss different conclusions from the information or data. 




0.614 


1.69 


Q59 


List positive (pro) and negative (con) reactions to the information. 




0.649 


1.32 


Scientific Thinking 


SciThi 


0.733 


1.93 


Q44 


Design ways to solve a problem. 




0.733 


1.52 


Q45 


Make guesses, predictions, or hypotheses. 




0.634 


2.15 


Q46 


Draw conclusions from science data. 




0.686 


2.11 


Q57 


Make prediction based on the information or data. 




0.678 


1.76 


Q60 


Reach conclusions or decisions based upon the information or data. 




0.695 


2.07 


Teacher Preparedness for Providing an Equitable Environment 


TPEQ 


0.811 


1.96 


Q98 


Teach students with physical disabilities. 




0.781 


1.40 


Q100 


Teach classes for students with diverse abilities. 




0.749 


2.08 


Q102 


Teach science to students from a variety of cultural backgrounds. 




0.810 


1.54 


Q103 


Teach science to students who have limited English proficiency. 




0.754 


1.66 


Q104 


Encourage participation of females in Science. 




0.799 


2.54 


Q105 


Encourage participation of minorities in Science. 




0.789 


2.52 
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Teacher Preparedness for Using Innovative Teaching Strategies 


TPIN 


0.829 


2.29 


Q91 


Use/manage cooperative learning groups in Science. 




0.778 


2.35 


Q92 


Integrate Science with other subjects. 




0.802 


2.30 


Q94 


Use of a variety of assessment strategies (including objective and 
open-ended formats). 




0.793 


2.35 


Q96 


Teach problem solving strategies. 




0.792 


2.35 


Q97 


Take into account students' prior conceptions about natural 
phenomena when planning curriculum and instruction 




0.807 


2.10 


Professional Collegiality 


PC 


0.599 


2.04 


Q114 


1 am supported by colleagues to try out new ideas in teaching 




0.582 


2.97 




science. 








Q116 


Science teachers in this school regularly share ideas and materials. 




0.534 


2.67 


Q117 


Science teachers in this school regularly observe each other 
teaching classes. 




0.547 


1.10 


Q120 


Most Science teachers in this school contribute actively to make 
decisions about the science curriculum. 




0.540 


1.99 


Q121 


1 have adequate time during the regular school week to work with 
my peers on science curriculum instruction. 




0.513 


1.45 


Use of Multiple Assessment Strategies 


MAS 


0.597 


1.89 


Q74 


Extended response item for which students must explain or justify 
solution. 




0.520 


2.38 


Q75 


Performance tasks or events (e.g., hands-on activities 




0.487 


2.43 


Q77 


Science projects 




0.554 


1.22 


Q78 


Portfolios 




0.536 


1.02 


Q79 


Systemic observations of students 




0.615 


2.40 


Standards 


STND 


0.579 


4.12 


Q80 


Your state's curriculum framework or content standards 




0.300 


4.43 


Q81 


Your district's curriculum frame work or guidelines 




0.401 


4.07 


Q85 


National science education standards 




0.694 


3.84 


Use of Educational Technology 


EDTEC 


0.552 


1.25 


Q40 


Use computers, calculators, or other technology to learn Science. 




0.496 


1.49 


Q62 


Use sensors and probes. 




0.548 


0.41 


Q63 


Collect data or information (e.g., using the internet). 




0.382 


1.51 


Q68 


Computer/lab interfacing devices included 




0.459 


1.53 
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Appendix C 



Methodology for Alignment Analysis of Instruction with Standards or Assessments as a 
Measure of Change in Instruction 

Central to the design of the experimental study underlying the Data on Enacted Curriculum 
(DEC) project are a series of alignment analyses designed to determine if instruction in phase 1 
(treatment) schools moves toward closer alignment with state standards and/or assessments than 
does instruction in phase 2 (control) schools. To conduct these analyses, two types of data are 
produced and analyzed using the Survey of Enacted Curriculum (SEC) two-dimensional content 
matrix (topic by expectations for learning). Teacher survey reports on content of instruction are 
compared with coded data on content of state standards and assessments. The data produce a 
common metric for quantitative analyses. 

The cells comprising the SEC content matrix (specific to math and to science) are used to code 
the content included in standards and assessments. Teachers use the same matrix to report on the 
content taught in class with their curriculum, making it possible to compute an objective measure 
of alignment. (A copy of the survey instrument is available at the CCSSO website, 
www.SECsurvey.org .) Data on the subject content of standards and assessments are coded by 
teams of four subject experts using SEC content framework based on established procedural rules 
and training techniques (Porter & Smithson, 2002). The alignment coding method has been found 
to have a high degree of inter-rater reliability among subject area specialist teams when applied to 
state assessments and standards (Porter, 2002). 

Other standards or assessment alignment procedures often have compared only topics of 
instruction. The primary reason for using a two-dimensional approach is that analyzing the 
intersection of both topics and expectations provides a powerful predictor of how well students 
learn, as demonstrated in research (Gamoran, Porter, Smithson, & White, 1997). 

Procedures in DEC project. The project included both content coding of state standards and 
assessments and surveys of teachers on instructional content. The description of instructional 
content was collected from teacher surveys, which were completed by 75 percent of the 
mathematics and science teachers in the DEC schools. Descriptions of state, local, and other 
assessments were collected as part of a content analysis workshop held in the fall of 2001 for 
DEC districts. 

The alignment statistic for each school or district is computed by comparing content data from 
two content descriptions (e.g., instruction and assessment). These data are reduced to a single 
index running from 0 to 1 , with 1 representing perfect agreement (or alignment) between the two 
descriptions and 0 indicating no agreement whatsoever. 

In considering alignment measures it is important to note the “target” used for aligning 
instruction. Typically, instruction will be aligned to content standards or to assessments. Which 



Data on Enacted Curriculum Study: Summary of Findings 



81 



target to select will largely depend upon the purpose for which the measures will be used. Content 
standards (if they are sufficiently detailed) are useful for providing a general description of 
desired instruction, as content standards cover a larger domain than is possible with assessment 
instruments. However, if the alignment measure is intended to be used as a tool for determining 
the effects of instruction on student achievement gains, then the assessment serves as a better 
target. Content standards also vary significantly by state, and some are likely to be more 
conducive to content analyses than others. For the DEC project thus far, only assessments have 
been content analyzed, in part because participating schools and districts tend to be more focused 
on assessments, upon which their accountability is based. 

Key role of alignment. Recently, federal and state policies in public education have required 
educators at all levels to focus their efforts in curriculum development and instructional 
improvement toward state standards in core subjects. The No Child Left Behind Act of 2001 
requires states to establish accountability systems that are based on statewide assessments which 
in turn are aligned with state standards, with a defined purpose of accountability as to ensure that 
schools focus improvement efforts on students with low performance. 

Currently educators and leaders at all levels are trying to improve alignment of policies as well as 
alignment of classroom instruction. The concept of alignment in education policy comes from the 
movement toward standards-based, system-wide education reform (Smith & O’Day, 1991; Porter 
& Smithson, 2001; National Research Council, 1995; National Council of Teachers of 
Mathematics, 2000). To improve education quality systemwide, policies governing K-12 
education, including curriculum, assessment, graduation, and teacher preparation, must be 
coherent and consistent — that is, aligned. 

A focus on alignment analysis is not just applicable to state policymakers; it is a powerful tool for 
local curriculum specialists, department heads, or classroom teachers. If poor and minority 
children are to receive a high quality, standards-based education — ultimately to reduce the 
achievement gap — then the instruction they receive must be aligned with the state content 
standards. Hence, a key element in understanding the impact of standards-based reform on 
student achievement is a measure of the alignment between the curricular content to which 
students are exposed and the content standards the state and district hope to implement. (For a 
summary of models, go to http://www.ccsso.org/proiects/Alignment Analysis .) 
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Interpreting Content Maps 

Content maps provide a three-dimensional representation of instructional content using a surface area chart which 
results in a graphic very similar to topographical maps. The grid overlaying each map identifies a list of topics areas 
(indicated by horizontal grid lines; see 1 below ) and six categories of cognitive expectations for students (indicated 
by vertical lines; see 2 below). The intersection of each topic area and category of cognitive expectation represents a 
measurement node (see 5 below). Each measurement node indicates a measure of instructional time for a given topic 
area and category of cognitive expectation based upon teacher reports. The resulting map is based upon the values at 
each of these measurement nodes. It should be noted that the spaces between each measurement node, that is the 
surface of the map, are abstractions and are not based upon real data, the image of the map is simply a computer 
generated graphic based upon the values for each intersecting measurement node. The map display is utilized to 
portray the third dimension (percent of instructional time; see 3 below) onto this grid utilizing shading and contour 
lines to indicate the percent of instructional time spent (on average across teachers) for each topic by cognitive 
expectation intersection. 



The increase (or decrease) in instructional time represented by each shaded band is referred to as the measurement 
interval (see 4 below). To determine the amount of instructional time for a given measurement node, count the 
number of contour lines between the nearest border and the node, and multiply by the measurement interval. 

The graphic at left below displays the three dimensional counterpart of the image represented by the content map 
displayed on the right. Both graphs indicate that Understanding Concepts related to Number Sense and Operations 
occupies the majority of time spent on grade four mathematics instruction (9% or more of instructional time over the 
course of a school year). 




Measurement Interval = 1% 
Data on Enacted Curriculum Study: Summary of Findings 
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NumberSense/ Properties 
/Relationships 



Measurement 



DataAnalysis/Probability/ 

Statistics 



Algebraic 



GeometricConcepts 



InstructionalTechnology 



Chart D-1 

Mathematics Curriculum Content 
Charlotte-Mecklenburg Schools 

Instruction / Assessment 



Charlotte-Mecklenburg 
Grade 6- 8 Teacher Reports (47) 




E 

E 
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Chart D-2 

Mathematics Curriculum Content 
School District of Philadelphia 

Instruction / Assessment 



School District of Philadelphia 
Grade 6- 8 Teacher Reports (24) 
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/Relationships 



Measurement 
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Chart D-3 

Mathematics Curriculum Content 
Chicago Public Schools 

Instruction / Assessment 



Chicago Public Schools 
Gr.6-8 Teacher Reports (19) 
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Chart D-4 

Mathematics Curriculum Content 
Winston-Salem/Forsyth County Schools 

Instruction / Assessment 



Winston-Salem/Forsyth 
County Schools 

Grade 6- 8 Teacher Reports (69) 
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Chart D-5 

Science Curriculum Content 
Miami-Dade County Public Schools 

Instruction / Standards 



Miami-Dade Public Schools 
Gr.6-8 Teacher Reports (82) 
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Chart D-6 

Science Curriculum Content 
Charlotte-Mecklenburg Schools 

Instruction / Standards 



Charlotte-Mecklenburg 
Grade 6- 8 Teacher Reports (18) 
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Chart D-7 

Science Curriculum Content 
School District of Philadelphia 

Instruction / Standards 



School District of Philadelphia Pennsylvania Grade 7 

Grade 6- 8 Teacher Reports (1 6) Science Content Standards 
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Chart D-8 

Science Curriculum Content 
Chicago Public Schools 

Instruction / Standards 



Chicago Public Schools 
Gr.6-8 Teacher Reports (10) 
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Chart D-9 

Science Curriculum Content 
Winston-Salem/Forsyth County Schools 

Instruction / Standards 



Winston-Salem/Forsyth 
County Schools 
Grade 6- 8 Teacher Reports (58) 
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